EschaLabs / Qwen3.8-27B-Escha-W2

Qwen3.8-27B-Escha-W2 — 2-bit quantized ( escha )

By Escha Labs Inc.

Escha-W2 is a 2-bit quantized build of Qwen3.8-27B . It holds the full 27B parameter count in 10.15 GB of weights — small enough that the whole model, its KV cache and a 64k context fit on a single 24 GB consumer card, with room left over.

On the three axes we measured it against a same-backend FP8 reference, this build is not measurably worse : it is ahead on commonsense reasoning, behind by exactly one question on GPQA-Diamond, and ahead on LiveCodeBench within that benchmark's own noise.

Contents

This repo holds only the model . The runtime that serves it lives in EschaLabs/escha-runtime-qwen3dense — an SGLang build with the decode kernels this format needs. That is the engine this model is served and measured on throughout this card.

Quickstart

python3.12 -m venv .venv && source .venv/bin/activate
pip install -U pip wheel

# PyTorch FIRST, and PINNED. The kernels are ABI-linked to it, and a bare
# "torch>=2.9" resolves to 2.11 with nothing to correct it — you would only find
# out later, via `undefined symbol: _ZN3c10...`.
pip install "torch==2.9.*" --index-url https://download.pytorch.org/whl/cu128

# The runtime wheel. It brings the SGLang build and the full dependency closure
# with it — do NOT also install sglang from PyPI, it will fight this one.
pip install -U "huggingface_hub[cli]"
hf download EschaLabs/escha-runtime-qwen3dense --include "sglang/*" --local-dir runtime
pip install ./runtime/sglang/escha-*.whl

# Weights (a separate repo; flat folder, no nested subdir)
hf download EschaLabs/Qwen3.8-27B-Escha-W2 --local-dir Qwen3.8-27B-Escha-W2

MODEL=./Qwen3.8-27B-Escha-W2 bash runtime/sglang/serve.sh

Sanity check before serving — all three must print True :

python -c "import torch, escha, sglang; print(torch.cuda.is_available(), \
hasattr(torch.ops.escha, 'escham_decode_gemv'), bool(sglang.__version__))"

( import sglang is part of the check on purpose — an earlier torch, escha -only version passed on a box that could not actually serve.)

Then, from another shell:

curl http://127.0.0.1:30000/v1/models

If generation is fluent but wrong — confident, well-formed nonsense — you are almost certainly on transformers < 5.8 , which loads this architecture with a silently different attention path. Upgrade before reporting a quality bug.

Connecting a client

opencode.json in this repo is a working provider block — drop it in and it points at the local server.

The launch script binds to localhost. If you set HOST=0.0.0.0 to reach it from another machine, set API_KEY at the same time; the server has no auth of its own.

Thinking mode

This is a thinking model. Toggle it through chat_template_kwargs — a top-level enable_thinking is ignored :

{
  "model": "escha-qwen38-27b-w2",
  "messages": [{"role": "user", "content": "..."}],
  "chat_template_kwargs": {"enable_thinking": true, "reasoning_effort": "xhigh"}
}

When thinking is on, the answer may arrive split across both reasoning_content and content — read both, or you will score empty strings. THINK=0 on the launch script defaults it off.

reasoning_effort — the knob most people should touch first

The default is xhigh , and it is the single biggest lever on how long an answer takes. It goes in chat_template_kwargs alongside enable_thinking , and it only applies while thinking is on.

Two things to know before you rely on it:

• It is a prompt, not a limit. Each level injects (or omits) one sentence of system instruction. It asks for shorter reasoning; nothing enforces it, and a hard problem can still produce a long chain at low . When you need a guarantee — a benchmark, an agent loop, anything with a timeout — use a thinking budget , which forces </think> after N tokens so an answer always gets produced: runtime cookbook → Bounded thinking .
• Any other value is a hard error. The template raises on anything outside those three, which surfaces as an HTTP 400 — not as a silent fallback to the default.

Tuning

serve.sh documents every knob at the top of the file. The ones that matter on a 24 GB card:

Per-architecture and per-VRAM launch recipes live in the runtime's "Running on your GPU" cookbook .

Verified configurations

Three tiers, each measured on the physical card named, 2026-08-20. These are the commands behind the numbers in Performance across GPUs .

# 24 GB, single user — the shipped defaults, full 64k context
MODEL=./Qwen3.8-27B-Escha-W2 bash sglang/serve.sh              # 4090: 67 tok/s bs1 · 18.1 GB
# on an RTX 3090 (or any sm_80/86) add ESCHA_ROUTE=blackwell   # 3090: 23.6 -> 40.7 tok/s bs1

# 24 GB, throughput — more streams, shorter context, graphs captured at every batch swept.
# MAXREQ/MAXMAMBA are what actually lift the stream ceiling: the shipped MEM/MAMBA_RATIO
# clamp a 24 GB card to 8-9 streams no matter what CUDA_GRAPH_BS says.
MODEL=./Qwen3.8-27B-Escha-W2 MEM=0.86 CTXLEN=32768 MAXREQ=32 MAXMAMBA=32 \
  CUDA_GRAPH_BS="1 2 4 8 12 16 24 32" bash sglang/serve.sh     # 4090: 649 tok/s @ 16 streams

# 32 GB (RTX 5090, sm_120) — triton attention is REQUIRED on consumer Blackwell
MODEL=./Qwen3.8-27B-Escha-W2 ATTN_BACKEND=triton MEM=0.85 CTXLEN=65536 MAXREQ=32 MAXMAMBA=32 \
  CUDA_GRAPH_BS="1 2 4 8 12 16 24 32" bash sglang/serve.sh     # 5090: 87.1 tok/s bs1 · 955 @ 16

Context and concurrency draw on one shared pool , and which half binds depends on your prompt length: with short prompts it is the recurrent state (~0.15 GB per stream, independent of context), with long prompts it is the KV pool. Either way, shortening CTXLEN alone does not buy you streams — raise MAXREQ / MAXMAMBA together with MEM , then check #running-req in the server log against the batch you actually asked for.

This checkpoint is text-only . The qwen3_5 config declares a vision tower, but the quantized weights contain none — it is in the quant ignore list, and the checkpoint has zero visual.* tensors. serve.sh sets SGLANG_VLM_TEXT_ONLY=1 so the tower is never instantiated. Do not send image inputs.

Requirements in detail

• GPU: NVIDIA sm_80 or newer (Ampere, Ada, Hopper, Blackwell). The wheel ships a fatbin covering sm_80/86/89/90/100/120.
• Driver only — no CUDA toolkit needed. If you have one, TRITON_PTXAS_PATH must point at the ptxas binary , not the directory containing it.
• glibc ≥ 2.28 (manylinux_2_28). RHEL 8+, Ubuntu 20.04+, Debian 10+.
• vm.overcommit_memory=1 avoids a large-allocation failure at load on memory-tight hosts. In a container /proc/sys is usually read-only; if you cannot set it, ample host RAM (≥32 GB free) is sufficient in practice — full campaigns have run at overcommit_memory=0 .
• transformers >= 5.8 . Ignore the transformers_version recorded in config.json — it reflects the machine that produced the export.

Format notes

• Each quantized projection stores escha_code plus the vectors escha_rin , escha_rout , escha_s_in , escha_s_out , and an escha_config header. The runtime derives each projection's code rate from the shape of its code tensor — bits: 2.0 in quantize_config.json is a family label, not the realized rate. The realized average over quantized projections is 2.469 bits/weight .
• 400 projections are coded; 449 tensors (norms, SSM A_log / dt_bias ) stay in fp16, and the embedding and output head are int8. Every coded projection also carries a small fp16 bias correction term, which the runtime applies as an output add.
• layer_meta in config.json is informational. The runtime does not read it.

Benchmarks

Two questions matter for a quantized build: how much quality did it cost , and what does it take to run . Quality first, then performance.

All quality numbers below were produced by serving this exact repo through the SGLang runtime linked above — not through a PyTorch reference path. Quantized models can score differently across backends, so a number measured on a backend nobody deploys is not a number worth publishing.

Quality vs FP8

The reference arm is the same base model served as FP8 on the same SGLang fork — which removes the backend itself as a variable. As a check that FP8 is a fair stand-in for the unquantized model, it scores 88.89 on GPQA-Diamond under our protocol against the 89.2 Qwen publishes for this model. Every arm below ran an identical protocol.

Commonsense-6 — full sets, no subsampling, thinking off:

Reasoning and code — thinking on, 28k-token budget:

How to read this. At 2.9× smaller than FP8, this build shows no measurable quality loss on the axes we tested. That is a stronger claim than it usually is at 2 bits, so here is the honest accounting of what each number can and cannot support:

• GPQA-Diamond −0.51 is one question out of 198 (175 vs 176 correct). Cap-hit rates are matched at 1/198, so no truncation confound — but the comparison is unpaired (we hold aggregates for the FP8 arm, not per-item answers) and the two arms ran on different cards. Read it as a tie, not as a deficit.
• LiveCodeBench +1.65 sits inside this build's own bootstrap σ of 2.51pp. Read it as a tie the quant happens to win. The FP8 arm's cap-hit rate was not recorded, so this pair has no truncation control.
• Commonsense-6 +1.29 is the one genuinely directional result, driven by the two ARC tasks (+6.65 and +3.20). One scoring choice to disclose: arc_easy is scored on raw acc for both arms , the other five on acc_norm . We made that call because acc_norm degenerates on arc_easy for this model family — length normalization, not the model, decides the answer — but it is a choice, it favors this build, and acc_norm on that one task would move the FP8 arm up. We hold aggregates rather than per-item answers for the FP8 arm, so we cannot publish the exact alternative number.

Evaluation protocol (reproduce these numbers)

Sampling for the thinking-on benchmarks: temperature 1.0, top_p 0.95, top_k 20, min_p 0, seed 42, reasoning_effort: "xhigh" (the default — see above ). The thinking budget is enforced server-side, closing the reasoning block so at least 4,096 tokens always remain for the answer.

Scoring reads only the text after the final reasoning block, uses multi-pattern answer extraction, and takes all prompts as the denominator — a truncated or unparseable generation counts as wrong, never as a skip.

Cross-box caveat. Commonsense-6 ran on an L40; GPQA-Diamond and LiveCodeBench ran on an RTX 4090. Within each benchmark both arms used identical prompts, budget, sampling and scorer, but the FP8 arm could not be run on the 4090 — it does not fit. Engine dtype therefore differs (fp16 vs bf16) and cannot be matched.

Performance across GPUs

Measured end-to-end on three consumer cards with the SGLang runtime, 2026-08-20 — the 4090 by us, the 5090 and 3090 by independent evaluators on their own hardware, from these public docs alone . Decode = what one user sees streaming ( 1000/TPOT ); peak throughput = total server output at the best concurrency.

[^amperoute]: Ampere owners: set ESCHA_ROUTE=blackwell . The kernel route auto-selects lovelace on sm_80/sm_86, and on an RTX 3090 that is the slower choice at batch 1 — 23.6 → 40.7 tok/s, a 1.72× speedup (TPOT 42.4 → 24.6 ms), with identical output on all sanity checks. The gain is batch-1-specific: parity at batch 2–16, and +25% again at batch 32 on short shapes. The 23.6 figure is what you get out of the box, so every 3090 row in the grid below is the un-forced route unless noted.

INT8 resolves ON automatically at ≤ 24 GB. On a 32 GB card it is off by default and worth turning on for single-user work: +7.2–7.4% decode at bs=1 on the 5090, measured as a paired A/B (83.7 → 89.9 tok/s at a 128-token prompt, 78.4 → 84.0 at 2k). Read the gain, not those absolutes — that session ran a few percent below the main sweep throughout. INT8=on does not free VRAM under a fixed MEM fraction: the ~1.2 GB saved on lm_head is recycled into the pool, so the KV cache grows instead (157,017 → 176,217 tokens, +12.2%). A user watching nvidia-smi for that 1.2 GB will conclude the flag did nothing.

This is not a VRAM ranking and not a generation ranking. The 24 GB 3090 and the 24 GB 4090 hold the same weights in the same pool, and the 3090 decodes at 61% of the 4090's rate even with the route fix (35% without it) while reaching only ~46% of its prefill rate. A 3090 has ~93% of a 4090's memory bandwidth, so a purely bandwidth-bound model would predict far better than either figure. Decode on this architecture is not purely bandwidth-bound at batch 1 — the small-M GEMV path matters, which is exactly why the route flag is worth 1.72× there. Corroborating that it is an occupancy effect: on the 3090, per-stream decode gets faster when a second stream is added (23.6 → 36.0 tok/s at 128/128, and a similar jump on every decode-heavy shape).

Serving — NVIDIA ISL/OSL grid, three GPUs

Single-stream decode tok/s by input/output shape, on the standard NVIDIA grid[^grid]. A dash means that evaluator did not run that shape — the three chose overlapping but not identical grids, so this is a union rather than a full matrix[^gridconv].

Mean TTFT at the same shapes, where a long prompt actually costs you something:

Single-stream decode barely moves with prompt length, on all three. From a 128-token prompt to a 5,000-token one it falls 11.5% on the 5090, 3.7% on the 4090 and 2.1% on the 3090; the 4090 still holds 90% of its short-prompt rate at a 20,000-token prompt. Decode is dominated by the weight read and the KV read is small beside it, so what a long prompt costs is TTFT, not tokens per second. The 5090's steeper decay is the one open question here — sm_120 is forced onto ATTN_BACKEND=triton while the others ran flashinfer, and a costlier attention path would produce exactly that signature (a penalty growing with KV size, short-prompt decode untouched). Settling it needs a triton vs trtllm_mha A/B on one 50-series card, which nobody has run.

For scale: fast reading is about 5 words/s (≈7 tok/s), so even the slowest card here decodes roughly 3× faster than you can read, and a 5090 about 12×.

Under concurrency

Aggregate output as streams are added:

The 4090 row is a 1,024-token output and the other two are 2,048 — both decode-heavy, but they are not the same cell, so read down a column, not across a row.

Throughput peaks at 16 streams on all three cards. An earlier build had a much sharper cliff past 16, because the fused decode kernel handled at most 16 rows per pass and anything above fell through to a path meant for prefill; that boundary now sits at 32 rows ( torch.ops.escha.escham_decode_gemv_max_m() ), worth about +20% at 24 streams on the 4090.

What is left is the shape in the table, and we still have not established why 16 beats 24 and 32 . What we can now say is what it is not : the peak-at-16 shape reproduces on sm_86, sm_89 and sm_120 — three architectures, two attention backends (flashinfer and triton), 24 and 32 GB, different pool sizes — and every run used captured CUDA graphs, so it is not the "uncaptured batch reads 15% low" artifact either. That rules out a card-specific or backend-specific cause and points at the scheduler or launch geometry. A memory-traffic model does not predict it. We would rather say that than publish a mechanism we have not verified.

Practically: 16 streams is the throughput sweet spot, and 32 is the ceiling of the fused path.

Three settings genuinely matter, all covered in the runtime cookbook :

• CUDA_GRAPH_BS must list your maximum batch size — and the shipped defaults do not reach 16. At MEM=0.72 MAMBA_RATIO=0.3 the recurrent-state pool admits only 8–9 concurrent streams on a 24 GB card, so max_running_requests clamps there and the 12 / 16 entries are silently dropped from the capture list. To actually serve more, raise MAXREQ / MAXMAMBA and MEM together — see Verified configuration . Capture is supported up to 32; above that the runtime refuses rather than bake in the wrong kernel.
• Recurrent state bounds concurrency at short context; the KV pool bounds it at long context. Every stream holds ~0.15 GB of ssm_state regardless of context length, which is what limits you with short prompts. With long ones the KV pool binds first: max_total_num_tokens (printed at startup) divided by your typical ISL+OSL is your real ceiling. If the server logs a #running-req below the batch you asked for, the surplus is queueing and your TTFT includes queue time — a 3090 at 2048/2048 asked for 32 streams, ran 18, and reported a 140 s mean TTFT that looks like a prefill catastrophe and is not.
• On Ampere, set ESCHA_ROUTE=blackwell if you serve one user at a time — see the footnote above; it is worth 1.72×.

[^grid]: Grid conditions: CUDA graphs on ( GRAPHS=1 , the serve.sh default), prefix caching off ( RADIX=0 , also the default here), output length pinned so every cell decodes its full OSL, and CUDA_GRAPH_BS extended to cover every batch swept. Reproducing these numbers without RADIX=0 and without the extended capture list will read low.

[^gridconv]: Three evaluators on three boxes with their own harnesses, so a word on what is comparable. Single-stream decode is the one metric all three report identically — per-stream rate, 1000 / TPOT — and it is quoted as measured, never derived. Their aggregate conventions do not match: some report a decode-window rate and some an end-to-end rate that amortizes prefill, which differ by ~4% at batch 1 on the same run, so the concurrency table is each card's own curve rather than a cross-card ranking. Two caveats those peaks inherit: short-output cells at batch understate steady decode (with a 128-token output the ramp in and out of the batch is most of the wall time — the 3090's 2048/128 peak of 68 tok/s is nearly all ramp), and at long inputs the KV pool cannot hold every requested stream , so some queue and the figure is pessimistic rather than a tuning miss. Prefill is quoted as ISL ÷ TTFT at batch 1 for all three cards; a batched prefill number is a different and much larger quantity, so do not mix them.

Licenses and attribution

This repository contains model weights only , released under Apache-2.0.

• Weights are a quantized derivative of Qwen/Qwen3.8-27B , Apache-2.0. See THIRD_PARTY_LICENSES/Qwen-LICENSE.txt .
• Tokenizer and chat template are carried over from the base model unmodified.

The runtime is distributed separately, in EschaLabs/escha-runtime-qwen3dense , with its own LICENSE and THIRD_PARTY_LICENSES/ covering SGLang and the other components it builds on — all permissive, no copyleft.