bottlecapai / ThinkingCap-Qwen3.6-27B-NVFP4

bottlecapai/ThinkingCap-Qwen3.6-27B-NVFP4

FP4 (NVFP4) quantization of bottlecapai/ThinkingCap-Qwen3.6-27B — capability of Qwen3.6-27B with 50% less thinking tokens on average, achieved by finetuning Qwen3.6-27B (Qwen Team, 2026) while preserving the original answer quality and style.

➡️ Full model description, evaluation results (multi-seed, statistically tested), recommended sampling params, and citation: see the main model card at bottlecapai/ThinkingCap-Qwen3.6-27B .

About this quantization

Weights are stored at 4 bits in NVIDIA's NVFP4 format (E2M1 values with per-group-of-16 FP8 scales) while activations stay in bf16 — weight-only NVFP4A16 (W4A16), produced data-free with llm-compressor in the compressed-tensors nvfp4-pack-quantized format that vLLM loads natively. ≈19 GB instead of ≈55 GB bf16 — ~2.9× smaller . Being weight-only (W4A16), it serves through vLLM's MarlinNvFp4LinearKernel on both Hopper (H100/H200) and Blackwell: the 4-bit weights unpack to bf16 for compute, so the decode win is memory bandwidth (≈2× bf16 tok/s), not FP4 tensor cores (native cutlass FP4 compute would require a W4A4 / FP4-activation checkpoint). The model's MTP (multi-token-prediction) head is preserved , so self-speculative decoding works (≈3.2 accepted tokens/step).

Kept in bf16: lm_head , the MTP head , the vision tower , and the two small Gated-DeltaNet input gates linear_attn.in_proj_a / in_proj_b (48-wide output, not a multiple of the 4-bit kernel's tile size and fused to width 96 at serve time, so quantizing them breaks loading on non-Blackwell GPUs; keeping them bf16 costs ~60 MB). The remaining group-quantizable Linear layers — MLP, self-attention, and the other linear-attention projections ( in_proj_qkv / in_proj_z / out_proj ) — carry the 4-bit weights.

Usage

vllm serve bottlecapai/ThinkingCap-Qwen3.6-27B-NVFP4

Speculative decoding (MTP)

The MTP (multi-token-prediction / NextN) head is kept in bf16, so vLLM can run self-speculative decoding for a decode speed-up — no separate draft model needed. Add --speculative-config when serving (requires vLLM ≥ 0.24.0):

vllm serve bottlecapai/ThinkingCap-Qwen3.6-27B-NVFP4 --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Speculative decoding is lossless — the output is identical to standard decoding. It accepts ≈3.2 drafts per verify step here, for a further ≈1.5× on top of the finetune's token savings (see below).

Served on vLLM ; SGLang cannot currently load weight-only NVFP4 for this architecture. For local llama.cpp / Ollama / LM Studio use, see the GGUF quantizations at bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF ; for near-lossless FP8 at half the memory (loads in vLLM and SGLang), see bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 .

Expected performance

Measured on our internal serving harness on 8× RTX PRO 6000 (Blackwell) with vLLM 0.25.0 , one configuration per GPU. Each cell is N=200 questions/dataset × 3 generation seeds over a fixed problem subset, batch size 16, sampled decoding (temperature 1.0, top_p 0.95, top_k 20); acc is the mean ± 95% CI across the seeds. Generation budgets are 24,576 tokens (MMLU-Pro) and 16,384 tokens (RealWorldQA).

FP4 matches the bf16 finetune's accuracy (all 95% CIs overlap) while decoding ≈2.3× faster (52.7 vs 22.6 tok/s, MMLU-Pro standard — the memory-bandwidth win of 4-bit weights via the Marlin kernel), and MTP self-speculative decoding (≈3.1–3.2 accepted tokens per verify step) adds a further ≈1.5× — stacking with the finetune's token savings to ≈4.7–8.4× faster per task than the unquantized base (MMLU-Pro 11.6 s vs 97.3 s; RealWorldQA 7.3 s vs 34.2 s).

All timings are measured per request during the eval. median tokens = median completion length (the finetune's lever); tok/s = per-request steady-state decode rate (prefill excluded via time-to-first-token); task s = measured end-to-end wall-clock per request under the batch-size-16 concurrency; speedup = task s of the base model in standard decoding ÷ task s of the row. unsloth NVFP4 and NVIDIA NVFP4 are community NVFP4 quants of the base model, shown for comparison; ThinkingCap-FP8 is our FP8 sibling ( bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 ).

NVFP4 kernel (important for reading tok/s ). On this hardware the vLLM NVFP4 kernel is chosen by the checkpoint's FP4 scheme , not a flag, so the NVFP4 rows are not on identical footing: this model and NVIDIA NVFP4 are weight-only W4A16 → MarlinNvFp4LinearKernel (4-bit weights unpacked for compute; a memory-bandwidth decode win), whereas unsloth NVFP4 is W4A4 (weights and activations FP4, mixed-precision ) → FlashInferCutlassNvFp4LinearKernel , the cutlass/flashinfer FP4 GEMM. vLLM has no cutlass path for weight-only W4A16 on sm_120, so those fall back to Marlin — a checkpoint-format property, bit-exact-ish and not a source of accuracy difference. On Hopper (H100/H200) there is no native FP4 compute at all, so every NVFP4 serves via Marlin weight-unpack; the cutlass FP4 tensor-core advantage (W4A4) shows mostly on compute-bound prefill / high batch and on datacenter Blackwell.

MMLU-Pro (reasoning)

RealWorldQA (vision)