avlp12 / Solar-Open2-250B-Alis-MLX-Dynamic

Solar-Open2-250B — Alis MLX Dynamic (DWQ) Builds

Built with Solar — this is a derivative of upstage/Solar-Open2-250B , distributed under the Upstage Solar License . A copy of the license ships in every branch.

MLX (Apple Silicon) port of Solar Open 2 (250B-A15B Hybrid-Attention MoE) in three quantization tiers, produced with alis-dwq — data-aware quantization with teacher-guided distillation (DWQ). Both distilled builds are fine-tuned against the Q8 reference under a per-slice acceptance gate, with an EN sentinel gate as the final safety net.

✅ Status: all three tiers final. Q v3 (4-bit/g32 + DWQ, lr 5e-7 × 3 rounds): EN loop cleared with teacher-level diversity (distinct4 0.399 vs teacher 0.411), code KLD better than its own base. F v2 (2.34 bpw, 68 GB): KLD 0.42 (−76% vs the legacy 2-bit path), all loops clean, EN sentinel ACCEPT. A fourth tier (M, 4-bit/g64) was trained twice (Q-teacher and Q8-teacher) but is not shipped — it lost every metric to Q v3 at the same memory tier.

Builds (branches)

One repo, three weights — pick a branch by machine memory:

Every build keeps all MoE router gates at 8-bit / g64 (48/48 layers) — router quantization was found to be the dominant quality risk, so only expert/attention weights are quantized per the tier recipe. Why no mid tier? An M build (4-bit/g64, 131–141 GB) was distilled twice (Q-teacher and Q8-teacher arms). The Q8 arm fixed its EN regression but still lost every metric to Q v3 at the same memory tier (≥160 GB machines), so it is retained locally as a control, not shipped.

Install & run

⚠️ Requires the solar_open2 architecture, which is not yet in PyPI mlx-lm (0.31.3). Stock pip install mlx-lm will fail with ValueError: Model type solar_open2 not supported . Two options:

# option A — the fork that adds solar_open2 (recommended)
# ⚠️ pin the solar-open2 branch — the fork's default branch (main) does NOT contain solar_open2
pip install "git+https://github.com/avlp12/mlx-lm.git@solar-open2"

Option B — keep stock mlx-lm and drop the bundled solar_open2.py (shipped in every branch) into .../site-packages/mlx_lm/models/ . Same file, same result.

# main = Q v3 (recommended)
mlx_lm.generate --model avlp12/Solar-Open2-250B-Alis-MLX-Dynamic \
  --prompt "안녕하세요, 간단히 자기소개를 해주세요." --max-tokens 512

Branch selection ( t96 = F v2, t384 = T Q8) uses the Python API — the CLI has no --revision flag:

from mlx_lm import load, generate

model, tok = load("avlp12/Solar-Open2-250B-Alis-MLX-Dynamic", revision="t96")   # F v2 (68 GB)
# model, tok = load("avlp12/Solar-Open2-250B-Alis-MLX-Dynamic", revision="t384") # T Q8 (266 GB)
print(generate(model, tok, prompt="Explain MoE routing in three sentences.", max_tokens=512, verbose=False))

CLI-only alternative: hf download avlp12/Solar-Open2-250B-Alis-MLX-Dynamic --revision t96 , then point --model at the downloaded snapshot directory.

F v2 LoRA adapters (t96 branch only): the DWQ compensator adapters ship un-fused in alis_adapters/ (rank 8). Download that folder and pass --adapter-path <local alis_adapters dir> to use them. All numbers in this card are measured without the adapters (same convention as the control arm), so a fused deployment may do better.

Chat, reasoning & sampling. The chat template ( chat_template.jinja ) ships in every branch; mlx_lm applies it automatically in chat/server mode (raw mlx_lm.generate with a plain string prompt does not — apply the template yourself). Client-side settings follow the base model card : temperature 1.0 / top_p 1.0 for both reasoning modes — avoid ~0.7-class temperatures, which measurably raise loop/runaway-thinking risk on quantized builds. reasoning_effort defaults to "high" (thinking ON); use "none" for a direct answer ( --chat-template-args '{"reasoning_effort":"none"}' on mlx_lm.server , or the apply_chat_template kwarg). Reasoning markers are dedicated tokens <|think:start|> / <|think:end|> (ids 130/131) — there is no </think> or "Final Answer" convention. Split reasoning from the final answer on <|think:end|> , and do not add it as a stop token: the answer comes after it, and termination is <|im:end|> (id 129, already in generation_config.json ). Budget max_tokens for thinking + answer together — thinking typically runs 0.5–2K tokens on chat prompts, so ≥4K total is recommended; an empty answer or a reply cut off mid-thought means the budget ran out inside the reasoning block. Server + LoRA adapters (F v2): fork revisions ≤ 9d67a0c silently dropped --adapter-path in mlx_lm.server — the adapter lookup ran after the model-name remap, always missed, and the server then served the bare 2-bit build. Fixed in 7937fda (branch solar-open2 ); update the fork, or pass "adapters": "<local alis_adapters dir>" in the request body as a workaround.

Hardware compatibility (memory planning)

Weights are loaded fully into GPU-wired unified memory; context costs extra on top. KV cache ≈ 48 KiB/token fp16 — only 12 of 48 layers (the softmax GQA layers, every 4th) hold a KV cache; the 36 KDA linear-attention layers carry a fixed-size recurrent state (~150 MiB total) that does not grow with context. → ~4.8 GB per 100K tokens · 256K ≈ 12 GiB · 1M ≈ 48 GiB.

Default GPU wired limit on macOS ≈ 80% of RAM ; raise it with sudo sysctl iogpu.wired_limit_mb=<MB> (revert by lowering again; needs admin).

Rule of thumb: pick the largest build whose weights fit your machine with the context you need — quality scales with bits (see the measured table below). DWQ training peak observed: 275 GB wired+compressed during Q v3 distillation (training-only; inference needs are far lower).

Note on the HF sidebar panel: the native Hardware compatibility widget indexes only the repo's root model ( main = Q v3, shown as "4-bit · MLX · 156 GB") — unlike GGUF repos, where every quant file gets its own row, an MLX repo gets a single row and branch builds ( t96 , t384 ) are not indexed. Use the table above for F/T hardware sizing.

Measured quality (vs Q8 reference, §2-10 eval, ref = T)

Q tier journey (disclosed): the 4-bit/g32 base shipped a greedy EN loop (len23). Attempt 1 (Q-dwq, legacy scalar gate) made it worse (len27) — the scalar gate's blind spot that motivated the per-slice gate. Attempt 2 (Q v2, gate + Q8 teacher, lr 1e-6 × 6 rounds) cleared the loop but collapsed EN diversity (distinct4 0.221) and regressed KLD +6.7% — over-distillation. Attempt 3 ( Q v3 , same gate/teacher but lr 5e-7 × 3 rounds) keeps the loop fix, restores EN diversity to 0.399 (97% of the Q8 teacher's 0.411) , brings KLD EN back to base level (+0.7%), and improves code KLD −5.1% below its own base . The residual trade-off is Korean KLD +8.4% vs base (0.10553 → 0.11438) — see Known Limitations.

F tier: the legacy path (uniform 2-bit/g128 base + clip + DWQ) failed the usability gate (KLD 1.23, all slices LOOPED) — a uniform 2-bit predicate destroys the KDA chokepoints and embed/head beyond DWQ recovery. F v2 re-quantizes with quant_predicate_builder (chokepoint 8b / embed+head 6b / attn+shared 4b / experts 2b, 578/578 per-layer entries verified) and distills from the Q8 teacher under a per-slice acceptance gate with rank-8 LoRA compensators. Result: KLD 1.73 (legacy base) → 1.23 (legacy DWQ) → 0.42 (F v2 + DWQ) , all loop probes clean, EN sentinel ACCEPT. The LoRA adapters ship separately ( alis_adapters/ ); the numbers above are measured without them (same convention as the legacy arm), so a fused deployment may do better.

DWQ training detail: 200 calibration samples (180 train / 20 valid), expert-routing coverage 99.8%; teacher = Q8 reference for all shipped builds ; deepest-first layer order, 3 rounds (Q v3) / 6 rounds (F v2); per-slice acceptance gate (EN ceiling ε=1%) + EN sentinel gate (paired, cap 1%).

Quantization quality (pre-distillation measurements)

Effective bit utilization (code-entropy audit, §2-2):

F clip requantization gate (T-dequant clipping before 2-bit requant):

Base model (bf16, official card numbers)

Solar Open 2 is Upstage's 250B-A15B open-weight model: Hybrid-Attention MoE — [Softmax, Linear×3] × 12 (only 12 of 48 layers hold a KV cache), NoPE on linear-attention layers, 321 experts (320 routed top-8 + 1 shared), vocab 196,608, 1M-token context , ~12T pre-training tokens, English/Korean/Japanese.

(Official numbers from the base model card ; quantized-build deltas are tracked in the KLD table above.)

Known Limitations

• Q v3 carries a disclosed Korean KLD regression (+8.4% vs its 4-bit/g32 base: 0.10553 → 0.11438) — the price of clearing the base's EN greedy loop while restoring EN diversity. English and code KLD are at or below base level; if your workload is Korean-critical and you can tolerate a known EN loop, prefer the T (Q8) tier or the base model.
• F v2 is a 2.34 bpw build: Korean KLD (0.75) is materially higher than the Q tier — expected at this bit budget; prefer Q v3 for Korean-heavy workloads. The DWQ LoRA adapters ship un-fused ( alis_adapters/ ); all reported numbers are without them.
• Reasoning-block runaway (root-caused, disclosed): an external integrator reported F v2 thinking that never closed (22.6K chars, no <|think:end|> ). This was traced to a serving bug, not the weights : fork mlx_lm.server ≤ 9d67a0c silently dropped --adapter-path , so that serve ran the bare 2-bit build without its DWQ LoRA compensators (fixed in 7937fda , see Install & run). With adapters actually applied, thinking terminated in 5/5 direct-path runs across both teams — at the official temp 1.0 / top_p 1.0 and at 0.7/0.95. Guidance stands regardless: official sampling, generous max_tokens , reasoning_effort="none" for short direct answers — and treat the adapters as part of the deployed F v2 configuration (a bare serve can ramble).
• Architecture support: solar_open2 is not yet in PyPI mlx-lm (0.31.3) — use the fork (pin @solar-open2 ; see Install & run) or the bundled solar_open2.py .
• Calibration coverage saturates at 200 samples (99.8% expert routing), but median tokens/expert (4,314) is ~19% of the 753B reference run; very rare experts may be under-distilled.
• Quantized builds inherit the base model's knowledge cutoff (2026-02) and language focus (EN/KO/JA).

License

Distributed under the Upstage Solar License (same as the base model). Per the license's derivative-model terms:

• Model name carries the "Solar" prefix ✅
• "Built with Solar" attribution displayed above ✅
• A copy of the Upstage Solar License is included in this repo ✅

Credits

• Base model: Upstage — Solar-Open2-250B ( Upstage Solar License ).
• MLX & mlx-lm : Apple ml-explore — solar_open2 architecture port (KDA hybrid-attention MoE, NoPE, sigmoid-gated GQA): Alis (avlp12) ( fork ).
• Mixed-precision DWQ quantization, per-slice acceptance gate + EN sentinel gate, F-tier quant_predicate_builder recipe (chokepoint 8b / embed+head 6b / attn+shared 4b / experts 2b): Alis (avlp12) ( alis-dwq ).

Citation

@misc{solar-open-2-2026,
  title={Solar Open 2 Technical Report},
  author={Upstage AI},
  year={2026},
  url={https://huggingface.co/upstage/Solar-Open2-250B}
}