0xSero / GLM-4.7-REAP-30

GLM-4.7-REAP-30

30% Expert-Pruned GLM-4 (358B -> 251B parameters)

A REAP-pruned version of zai/glm-4.7 (GLM-4.7), reducing the model from 358B to ~251B parameters by pruning 30% of MoE experts while preserving model quality.

Acknowledgments

This work was made possible by:

• Prime Intellect - Compute sponsorship (8x H200 cluster)
• Cerebras - REAP methodology ( arXiv:2505.20877 )
• Intel - AutoRound quantization framework

Model Details

Related Models

Usage

Basic Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "0xSero/GLM-4.7-REAP-30"

tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

messages = [{"role": "user", "content": "Write a Python function to check if a number is prime."}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
inputs = inputs.to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id
)
response = tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True)
print(response)

With vLLM (Recommended for Production)

from vllm import LLM, SamplingParams

llm = LLM(
    model="0xSero/GLM-4.7-REAP-30",
    tensor_parallel_size=8,  # Use all 8 GPUs
    trust_remote_code=True,
    dtype="bfloat16"
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = llm.generate(["Write a Python function to check if a number is prime."], sampling_params)
print(outputs[0].outputs[0].text)

REAP Methodology

REAP (Router-Experts Activation Pruning) is a state-of-the-art MoE pruning technique developed by Cerebras that identifies and removes the least important experts based on their activation patterns.

How REAP Works

                    Original GLM-4.7                          REAP-Pruned
                    ================                          ============

    Input -----> [Router] -----> [Expert 1]  (high activation) --> KEEP
                         |-----> [Expert 2]  (low activation)  --> PRUNE
                         |-----> [Expert 3]  (high activation) --> KEEP
                         |-----> ...
                         |-----> [Expert 160] (low activation) --> PRUNE

    Result: 160 experts -> 112 experts ({100-COMPRESSION_PCT}% retained)

REAP Algorithm Steps

1. Calibration Phase : Run representative samples through the model
2. Saliency Scoring : Compute importance scores for each expert:
   • Router assignment frequency (how often selected)
   • Activation magnitude weighted by routing probability
   • Angular distance clustering (identifies redundant experts)
3. Pruning : Remove the lowest-scoring experts (30% removed)
4. Router Adjustment : Update router weights for remaining experts

Calibration Dataset

We used a diverse calibration mix optimized for code and agentic tasks:

Combined dataset: 0xSero/glm47-reap-calibration-v2

REAP Configuration

compression_ratio: 0.30
distance_measure: angular
seed: 42
samples: 1360
model_max_length: 2048
prune_method: reap

Reproduce This Model

Prerequisites

# Clone REAP repository
git clone https://github.com/Cerebras/reap
cd reap
pip install -e .

# Or use our fork with GLM-4.7 support
git clone https://github.com/0xSero/reap

Run REAP Pruning

# Full calibration run (~5 hours on 8x H200)
python src/reap/prune.py \
    --model-name zai/glm-4.7 \
    --dataset-name 0xSero/glm47-reap-calibration-v2 \
    --compression-ratio 0.30 \
    --prune-method reap \
    --seed 42 \
    --samples_per_category 1360 \
    --model_max_length 2048 \
    --distance_measure angular \
    --record_pruning_metrics_only true \
    --output_file_name observations_1360_angular-seed_42.pt

# With observation reuse (instant, <5 minutes)
# If you already have the observations file from a previous run:
python src/reap/prune.py \
    --model-name zai/glm-4.7 \
    --compression-ratio 0.30 \
    --load_observations observations_1360_angular-seed_42.pt \
    --prune-method reap \
    --seed 42

Observation Reuse

The key insight: REAP's calibration phase computes per-expert saliency scores that are independent of the compression ratio . Once computed, you can instantly generate models at any pruning level:

# observations_1360_angular-seed_42.pt contains:
# - Expert activation statistics per layer
# - Router weight distributions
# - Angular distance matrices
# Size: ~19MB for GLM-4.7

# Generate multiple variants instantly:
for ratio in [0.30, 0.35, 0.40, 0.50]:
    run_reap(compression_ratio=ratio, load_observations="observations.pt")

Hardware Requirements

Benchmarks

Coming soon - benchmarks are in progress

Citation

If you use this model, please cite:

@article{jones2025reap,
  title={REAP: Router-Experts Activation Pruning for Efficient Mixture-of-Experts},
  author={Jones, et al.},
  journal={arXiv preprint arXiv:2505.20877},
  year={2025}
}

@misc{glm47reap,
  title={GLM-4.7-REAP: Expert-Pruned GLM-4 Models},
  author={0xSero},
  year={2026},
  howpublished={\url{https://huggingface.co/0xSero/GLM-4.7-REAP-30}}
}

Limitations

• Requires multi-GPU setup for inference (~470GB VRAM)
• Some capability degradation vs full model (expected with 30% pruning)
• Calibrated primarily on code/agentic tasks; may have reduced performance on other domains
• Use W4A16 variant for lower VRAM requirements

License

Apache 2.0 (inherited from base GLM-4 model)

Built with REAP by Cerebras | Sponsored by Prime Intellect