DJLougen / LFM2-24B-A2B-REAP

LFM2-24B-A2B-REAP — Expert-Pruned LiquidAI LFM2

Router-weighted Expert Activation Pruning (REAP) applied to LiquidAI/LFM2-24B-A2B , reducing experts from 64 → 32 per MoE layer (50% expert reduction).

What is REAP?

REAP (Router-weighted Expert Activation Pruning) analyzes the router gate weights in Mixture-of-Experts models to identify which experts contribute least to model outputs. Experts with the lowest router weight magnitudes — indicating they are rarely or weakly selected during inference — are pruned. The remaining experts retain their original weights, preserving model quality while significantly reducing size.

Model Details

Architecture

LFM2-24B-A2B is a hybrid architecture combining:

• 30 convolutional layers (short convolution with gated projections)
• 10 full attention layers (GQA with 32 heads, 8 KV heads)
• 38 MoE FFN layers (layers 2-39, with top-4 routing from 64 experts)
• 2 dense FFN layers (layers 0-1, dense feed-forward with intermediate_size=11776)

The MoE routing uses:

• Softmax-normalized top-4 selection
• Expert bias terms (learnable per-expert additive bias)
• Routing scaling factor of 1.0

Available Quantizations

GGUF (llama.cpp)

MLX (Apple Silicon)

Usage

llama.cpp

# Q4_K_M (recommended for most use cases)
llama-cli -m LFM2-24B-A2B-REAP-Q4_K_M.gguf \
  --jinja -ngl 99 -fa \
  --temp 0.7 --top-k 40 --top-p 0.95 \
  -c 32768 -n 8192

# BF16 (full precision, needs ~32GB RAM)
llama-cli -m LFM2-24B-A2B-REAP-BF16.gguf \
  --jinja -ngl 99 -fa \
  -c 32768 -n 8192

MLX (Apple Silicon)

from mlx_lm import load, generate

model, tokenizer = load("DJLougen/LFM2-24B-A2B-REAP-MLX-4bit")
response = generate(model, tokenizer, prompt="Hello!", max_tokens=512)

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "DJLougen/LFM2-24B-A2B-REAP",
    torch_dtype="bfloat16",
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("DJLougen/LFM2-24B-A2B-REAP")

REAP Methodology

Expert Importance Scoring

For each MoE layer, expert importance is computed from the router gate weight matrix W_gate ∈ R^{num_experts × hidden_size} :

importance(expert_i) = ||W_gate[i, :]||_2

Experts with the lowest L2 norm of their router weight vectors are pruned. This metric captures how strongly the router can direct tokens to each expert — experts with small weight norms are rarely or weakly selected.

Pruning Process

1. Extract router weights from all 38 MoE layers
2. Compute L2 importance for each of 64 experts per layer
3. Select bottom 32 experts (per layer, independently) for removal
4. Slice expert tensors to remove pruned experts' parameters
5. Update router gate to match reduced expert count
6. Update expert bias vector to match reduced expert count
7. Update model config ( num_experts: 64 → 32 )

Tensors Modified

Per MoE layer (38 layers × 5 tensor types = 190 tensors pruned):

What's Preserved

• All dense layer weights (layers 0-1)
• All attention/conv layer weights
• All normalization parameters
• Token embeddings
• Tokenizer and chat template
• Active parameter count (2.3B) remains the same since top-4 routing is preserved

Evaluation

Inference Performance (DGX Spark, Q4_K_M)

Quality

The REAP'd model produces coherent, correct outputs. Perplexity benchmarks to be added.

Hardware Requirements

Reproduction

# 1. Run REAP pruning
python reap_lfm2.py \
    --input LFM2-24B-A2B-BF16.gguf \
    --output_dir ./lfm2-reap \
    --target_experts 32

# 2. Quantize GGUF variants
llama-quantize LFM2-24B-A2B-REAP-BF16.gguf LFM2-24B-A2B-REAP-Q4_K_M.gguf Q4_K_M
llama-quantize LFM2-24B-A2B-REAP-BF16.gguf LFM2-24B-A2B-REAP-Q5_K_M.gguf Q5_K_M
llama-quantize LFM2-24B-A2B-REAP-BF16.gguf LFM2-24B-A2B-REAP-Q6_K.gguf Q6_K
llama-quantize LFM2-24B-A2B-REAP-BF16.gguf LFM2-24B-A2B-REAP-Q8_0.gguf Q8_0

# 3. Create MLX versions
python prune_safetensors_for_mlx.py --expert_mapping expert_mapping.json --output_dir mlx-reap-hf
mlx_lm.convert --hf-path mlx-reap-hf --mlx-path mlx-reap-4bit -q --q-bits 4
mlx_lm.convert --hf-path mlx-reap-hf --mlx-path mlx-reap-8bit -q --q-bits 8

Citation

If you use this model, please cite the original LFM2 work:

@article{liquidai2025lfm2,
    title={LFM2: Liquid Foundation Models 2},
    author={Liquid AI},
    journal={arXiv preprint arXiv:2511.23404},
    year={2025}
}

License

This model inherits the LFM 1.0 License from the base model.

---

Created by DJLougen using REAP (Router-weighted Expert Activation Pruning)