0xSero / GLM-5.1-555B

Support this work → · X · GitHub · REAP paper · Cerebras REAP

GLM-5.1-555B

REAP-pruned zai-org/GLM-5.1 .

At a glance

Which variant should I pick?

DO NOT USE THIS MODEL FOR ANYTHING SERIOUS.

This checkpoint has not been benchmarked, validated, or tested for coherence . It may produce garbage, repetitive loops, incoherent text, or complete nonsense . Treat it as a broken artifact until proven otherwise .

GLM-5.1 — 25% Expert Pruned (REAP)

This is a 25% expert-pruned version of zai-org/GLM-5.1 using the REAP method (Relative Expert Activation Pruning).

Why This Might Be Broken

1. Partial calibration data — The saliency scores used to select experts for removal were computed from only ~35% of the planned 22,000-sample calibration corpus. Expert importance rankings may be inaccurate.

2. No quality testing whatsoever — Zero benchmarks have been run. No coherence check. No perplexity measurement. No human evaluation. The model could produce degenerate output for all we know.

3. Aggressive prune ratio — Prior experiments with GLM-family models at similar or higher prune ratios resulted in complete output collapse (repetitive text, broken reasoning, junk logits). The 50% checkpoint in particular is very likely broken based on prior GLM-5 evidence.

4. DSA architecture sensitivity — GLM-5.1 uses Dynamic Sparse Attention with learned indexer weights. The interaction between pruned expert routing and the DSA indexer has not been validated.

5. refusal_contrast_reap without preserve guards — The pruning was done using refusal_contrast_reap selection without preserve_super or preserve_outlier guardrails, which in prior GLM-5 experiments led to output collapse at high prune ratios.

What This Is Useful For

• Research only. Specifically:
  • Studying REAP expert saliency patterns in GLM-5.1
  • Comparing prune-ratio robustness across architectures
  • Running your own coherence/benchmark evaluations
  • Investigating MoE collapse behavior

How to Load

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "0xSero/GLM-5.1-555B",
    device_map="auto",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("0xSero/GLM-5.1-555B", trust_remote_code=True)

# IMPORTANT: GLM-5.1 is a thinking/chat model. Use the chat template.
messages = [{"role": "user", "content": "Hello"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
out = model.generate(inputs.to(model.device), max_new_tokens=128)
print(tokenizer.decode(out[0]))

Pruning Method

REAP (Relative Expert Activation Pruning) removes MoE experts by measuring their relative activation patterns during a calibration pass. Experts with the lowest saliency scores (combined REAP signal + frequency weighting) are removed layer-by-layer, keeping top-8 routing unchanged so the active-parameter budget per token stays the same.

Sibling Checkpoints

All three are untested. The 25% checkpoint is the most likely to be coherent.

License & citation

License inherited from the base model.

@misc{lasby2025reap,
  title  = {REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression},
  author = {Mike Lasby and Ivan Lazarevich and Nish Sinnadurai and Sean Lie and Yani Ioannou and Vithursan Thangarasa},
  year   = {2025}, eprint = {2510.13999}, archivePrefix = {arXiv}
}

Sponsors

Made possible by NVIDIA · TNG Technology · Lambda · Prime Intellect · Hot Aisle .