EntityDeletr / North-Mini-Code-1.0-EAGLE3-GGUF

Quantized version of AlexWortega/North-Mini-Code-1.0-EAGLE3 .

Their model card is pasted as is below.

Files:

• model.safetensors - original unquantized safetensors
• model.gguf - unquantized bf16 GGUF
• North-Mini-Code-1.0-EAGLE3.gguf - GGUF quantized to Q5_K_M

North-Mini-Code-1.0 — EAGLE-3 draft head

EAGLE-3 draft model for CohereLabs/North-Mini-Code-1.0 ( cohere2_moe , 30B/3B MoE, 49 layers), trained with SpecForge (offline) for lossless speculative decoding.

• Draft : 1 Llama-style decoder layer, hidden 2048, FFN 12288, draft_vocab 32000 (freq-reduced from 262144).
• Aux hidden-state layers : [1, 23, 45].
• Training : offline, ~8.3k code-instruction samples (magicoder-evol-instruct), 10 epochs, lr 1e-4.
• Offline held-out acceptance (Σ over 7 positions): τ = 4.25 (pos-0 acc 0.71).

Serving in vLLM

Needs vLLM main + --hf-overrides '{"first_k_dense_replace":1}' , and a patch adding the EAGLE3 interface ( SupportsEagle3 ) to cohere2_moe.py . See repo notes.

vllm serve CohereLabs/North-Mini-Code-1.0 \
  --speculative-config '{"method":"eagle3","model":"<this-repo>","num_speculative_tokens":5}' \
  --hf-overrides '{"first_k_dense_replace":1}' \
  --reasoning-parser cohere_command4 --tool-call-parser cohere_command4 --enable-auto-tool-choice

Note: this draft was trained offline on HuggingFace-transformers hidden states; real vLLM acceptance is modest (~1.28) due to train/serve hidden-state representation mismatch. For best speedup, retrain online in vLLM/SpecForge so the draft matches serving-time hidden states.