saricles / MiniMax-M2.7-NVFP4-GB10

MiniMax-M2.7-NVFP4-GB10

Custom GB10 NVFP4 quantization of MiniMaxAI/MiniMax-M2.7 (230B, 256 MoE experts, top-K=8) targeted at NVIDIA DGX Spark (GB10) and Blackwell-family hardware. 130.6 GB on disk, down from the 230.2 GB official FP8 release.

Model Details

Quantization Details

• Method: Post-training quantization via NVIDIA TensorRT Model Optimizer ( nvidia-modelopt )
• Scheme: mtq.NVFP4_DEFAULT_CFG + GB10-tuned disable list applied post-calibration
• Calibration Dataset: HuggingFaceH4/ultrachat_200k ( train_sft split)
• Calibration Samples: 64
• Max Sequence Length: 2048 tokens
• Preserved in BF16 (ignore list): lm_head , *block_sparse_moe.gate (MoE router gate, not per-expert gates)
• Hardware Used: Hugging Face Jobs, 8× NVIDIA A100 80 GB
• Recipe script: quantize-nvfp4-gb10.py — env-var-configurable; see the file header for the MoE-expert amax gotcha and how to adapt for other architectures.

Performance (2× NVIDIA DGX Spark — GB10, 128 GB each)

Benchmarked with llama-benchy 0.3.3, 3 runs per scenario, TP=2 over ConnectX-7 RoCE, 192K context window.

Decode throughput is stable at ~26 tok/s across all prompt lengths — characteristic of a dual-node tensor-parallel deployment where decode is bound by per-step inter-node communication rather than compute. Prefill scales cleanly with prompt length as the per-step batch amortizes fixed overhead.

Running on 2× DGX Spark (Tensor Parallel)

At 130.6 GB this model does not fit in a single DGX Spark's 128 GB unified memory. It is intended to run with tensor-parallel-size=2 across two Sparks connected via their ConnectX-7 200 GbE link, orchestrated by Ray. The community reference container is eugr/spark-vllm-docker .

Environment variables

These are the ones that matter for NVFP4 on GB10 and for NCCL over the ConnectX-7 link. Interface names ( enp1s0f0np0 , rocep1s0f0 , roceP2p1s0f0 ) may differ on your hardware — verify with ip -br link and ibdev2netdev .

# NVFP4 kernel selection (validated on GB10 SM 12.1)
VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass
VLLM_USE_FLASHINFER_MOE_FP4=0
SAFETENSORS_FAST_GPU=1
OMP_NUM_THREADS=8
TORCHINDUCTOR_MAX_AUTOTUNE=0

# NCCL + Gloo + Ray — route everything over the RoCE interface
NCCL_SOCKET_IFNAME=enp1s0f0np0
GLOO_SOCKET_IFNAME=enp1s0f0np0
TP_SOCKET_IFNAME=enp1s0f0np0
UCX_NET_DEVICES=enp1s0f0np0
NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0    # both halves for full 200 Gb/s
NCCL_IB_DISABLE=0
NCCL_IGNORE_CPU_AFFINITY=1

# Per-node
VLLM_HOST_IP=<this node's RoCE IP>

Ray cluster

On the head node:

ray start --head --port=6379 \
  --node-ip-address=<HEAD_ROCE_IP> \
  --disable-usage-stats

On the worker node:

ray start --address=<HEAD_ROCE_IP>:6379 \
  --node-ip-address=<WORKER_ROCE_IP> \
  --disable-usage-stats --block

vLLM server (head node, after Ray reports 2 GPUs)

vllm serve /models/MiniMax-M2.7-NVFP4-GB10 \
  --host 0.0.0.0 --port 30000 \
  --served-model-name minimax-m2.7 \
  --tensor-parallel-size 2 \
  --distributed-executor-backend ray \
  --gpu-memory-utilization 0.85 \
  --max-model-len 196608 \
  --kv-cache-dtype fp8_e4m3 \
  --attention-backend flashinfer \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-seqs 64 \
  --max-num-batched-tokens 8192 \
  --trust-remote-code \
  --enable-auto-tool-choice --tool-call-parser minimax_m2 \
  --reasoning-parser minimax_m2_append_think \
  --compilation-config '{"cudagraph_mode":"none","inductor_compile_config":{"combo_kernels":false,"benchmark_combo_kernel":false,"max_autotune":false,"max_autotune_gemm":false}}'

First boot loads the weights and runs JIT compilation — plan for 10–15 minutes before /v1/models responds.

Test it

curl http://<HEAD_HOST>:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "minimax-m2.7",
    "messages": [{"role": "user", "content": "Hello!"}],
    "temperature": 1.0, "top_p": 0.95, "top_k": 40, "min_p": 0.01,
    "max_tokens": 512
  }'

Gotchas

• --gpu-memory-utilization 0.85 is the safe default for TP=2. Activation and inter-node comm buffers grow under tensor parallel; raise cautiously.
• Both RoCE halves must be in NCCL_IB_HCA , or you get 100 Gb/s instead of 200.
• cudagraph_mode: none in --compilation-config is load-bearing for MiniMax MoE on GB10 — CUDA graphs deadlock on this architecture. Leave torch.compile otherwise enabled.
• Do not pass --enable-expert-parallel — uneven per-node memory on MoE, breaks under current kernels.
• If NCCL_DEBUG=INFO shows NET/Socket instead of NET/IB/0 + NET/IB/1 , NCCL fell back to TCP — recheck interface names and NCCL_IB_HCA .

Recommended Sampling Parameters

Per MiniMax documentation :

{
  "temperature": 1.0,
  "top_p": 0.95,
  "top_k": 40,
  "min_p": 0.01
}

Target Hardware

Quantized for and tested on NVIDIA DGX Spark (GB10, 128 GB unified memory, 221 GB/s bandwidth). Should work on other Blackwell-class GPUs with NVFP4 tensor-core support. On Hopper-class hardware (H100/H200) the model will load and run, but the ignore-list was tuned for Blackwell and will leave some performance on the table.

If you only have one DGX Spark

At 130.6 GB this model does not fit in a single Spark's 128 GB unified memory — it requires 2× Spark with tensor parallelism. If you have only one Spark, consider the REAP-pruned variant: saricles/MiniMax-M2.7-REAP-172B-A10B-NVFP4-GB10 (98.9 GB, single-node deployment, 25% fewer MoE experts with the capabilities you most likely care about preserved via agentic calibration).

Acknowledgments

• Base model by MiniMax
• Quantization tooling: NVIDIA TensorRT Model Optimizer
• GB10 quantization profile guidance: Scott Glover (scottgl)
• Multi-Spark runtime tuning: the eugr/spark-vllm-docker project and the NVIDIA Developer Forum community