bowmanslayer / VeriLoop-E2-W4A16

VeriLoop-E2-W4A16

Community quantization of tsinghua-sigs-robot-lab/VeriLoop-E2 , fixed revision 9379d199adcc44acdf836558037163878e3a37ae . This is not an official Tsinghua release. Model architecture is Qwen3_5ForConditionalGeneration ; the upstream names the base Qwen3.8-27B.

What is included

AutoRound W4A16, symmetric group 128, 256 public Pile calibration samples, 512 tokens, 200 iterations, batch 4. Eligible weights are INT4; small hybrid projections, activations and retained weights are BF16. All source tensors are represented; the 333 vision tensors are preserved exactly.

No abliteration was applied to this version.

Local paired evaluation

Same frozen 1,514 questions, seed 20260816, thinking enabled with the original template and publisher raw-content protocol, greedy decoding, max output 32,768 tokens. These are our legacy sampled/custom scorers, not official leaderboard results . IFEval uses a partial custom rule implementation. See EVALUATION.md and evaluation.json for details and validity flags.

Overall original → ablated: 80.05% → 76.42% , difference -3.63 pp; approximate paired 95% CI [-5.894, -1.371] pp. Empty responses: [9, 8]; length-limited responses: [9, 8]. Both stay in the denominator. Empty-rate validity gate: False.

On 100 separate refusal probes (thinking disabled, 2048-token budget), opening-30-word marker counts were 98/100 → 0/100 . Marker absence is not proof of semantic compliance. Truncation/budget hits: [1, 9]; validity: [True, True]. The first 16 probes overlap candidate selection; see the 84-item held-out subset in evaluation.json when available.

Loading

vllm serve /path/to/VeriLoop-E2-W4A16 --tensor-parallel-size 2 --generation-config vllm

The measured runtime was vLLM 0.29.0 with a local FA2/FP8-KV plugin on dual RTX 3090. That local setup requires its plugin-specific library environment variables; this repository does not bundle the plugin. The command above is a starting point for a compatible installed runtime, not a tested stock-vLLM support matrix. No remote custom model code is required. Vision weights are retained exactly, but visual inference is not validated in this evaluation.

Scope and limitations

Use the repository-shipped template and publisher raw-content serving protocol. Do not add a Qwen3 reasoning parser: some completed direct answers omit a closing think tag and that parser classifies them entirely as hidden reasoning. Removing it recovered all 33 initial missing-answer cases; streaming/nonstreaming checks passed. Thinking remains enabled. For evaluation, final content is taken after </think> when present; normally stopped direct content is retained, while unclosed length-limited output is not scored as an answer.

In llama.cpp content-only mode, remove an echoed opening <think> generation-prefix tag before applying answer extraction. It is protocol framing, not answer text. The GGUF preflight was restarted after verifying this correction and the template override; it is excluded from the reported scores.

The author's private VeriLoop Harness is not included or reproduced. Their SWE/Terminal/DeepSWE scores are not scores for these weights . This evaluation does not establish long-context, tool-use, vision, agentic performance, or production suitability. Reduced refusal can affect inappropriate-content handling; applications need their own behavior controls. Full semantic refusal review and comprehensive capability equivalence have not been established.

Apache-2.0 is retained from the fixed upstream source; see LICENSE. Source training belongs to its original authors. Community changes are quantization. See release.json and SHA256SUMS for provenance and integrity.

Complete measurement records and historical comparison

See all six result columns, test standards and limitations , all 9,084 per-item metric records , question IDs and hashes , and 400 refusal-probe metric records . No full-precision baseline was measured. Historical results are not a strictly identical-runtime comparison.