cnuland / legunas21-custom

Laguna-S-2.1 on AMD Strix Halo (llama.cpp + Vulkan) — deployment kit

Working configuration, tuning notes, and benchmark data for running poolside/Laguna-S-2.1 (118B MoE, ~8B active) on an AMD Strix Halo iGPU (Radeon 8060S, RDNA3.5, unified memory) under OpenShift, using the unsloth UD-IQ4_XS GGUF (57.5 GB) fully offloaded via the llama.cpp Vulkan backend (RADV).

Hardware: 16 GiB VRAM carve-out + ~55 GiB GTT on a 114 GiB node — unified memory is what makes -ngl 99 possible for a 57 GB model on an iGPU.

Files

Benchmarks (2026-07-23, worker node: Strix Halo, 114 GiB RAM)

Decode = 512-token code generation, temp 0. Prefill = 4.2K-token prompt.

Real-world caveat (why the default ships spec-off): on a realistic chat-templated coding task (Go middleware + tests, ~2-5K tokens out), DFlash Q4/n=4 measured 25.3 t/s vs 30.4 t/s without spec . Chat output mixes prose with code and drops draft acceptance below break-even on bandwidth-shared unified memory. DFlash wins only on near-pure code streams (raw completion / FIM). Flip it on per-deployment when that is your workload:

-md laguna-s-2.1-DFlash-Q4_K_M-76t.gguf --spec-type draft-dflash \
--spec-draft-n-max 4 -ctkd f16 -ctvd f16 -ngld 99

Hard-won details

1. DFlash needs poolside's llama.cpp fork ( poolsideai/llama.cpp , branch laguna ). Upstream (≥ b10087 has the Laguna arch, tested through b10103) fails on any DFlash draft with done_getting_tensors: wrong number of tensors; expected 76, got 69 — the draft borrows token_embd / output.weight from the target and upstream's decoder contract is incomplete. Apply speculative-cmath.patch to build the fork with GCC 15 / Ubuntu 26.04.
2. unsloth GGUF metadata bug : the quant ships laguna.rope.scaling.yarn_attn_factor = 1.4852 , which llama.cpp double-applies (it derives mscale itself — poolside corrected their own GGUFs to 1.0 on 2026-07-23). Fix at runtime, no re-download: --override-kv laguna.rope.scaling.yarn_attn_factor=float:1.0
3. Sampling : use the model card's --temp 0.7 --top-p 0.95 . The shipped generation_config.json (1.0/1.0/top_k 20) disagrees with the card and measurably increases fabrications (community eval: 2.4% → 0.8% of grounding runs).
4. Draft KV must stay f16 ( -ctkd f16 -ctvd f16 ) — quantized draft KV collapses acceptance below 1%.
5. --spec-draft-n-max 4 , not the 15 from poolside's early example (their updated rec is 7; on bandwidth-limited hardware 4 measured best; 7 was a net loss).
6. Flash attention ON is 2.6× decode / 1.7× prefill vs off on RDNA3.5 Vulkan. The RDNA3.5 FA crashes reported against PR #25165 are on the ROCm path; the Vulkan/RADV path ran clean here.
7. Clients should send max_tokens ≥ 8k — the model thinks long and returns empty content when the budget is eaten by reasoning.
8. Vulkan env that works on Strix Halo (device plugin exposes amd.com/gpu ): AMD_VULKAN_ICD=RADV , VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/radeon_icd.json , HIP/ROCR disabled.

License

OpenMDW-1.1, inherited from Laguna S 2.1. The DFlash GGUFs are unmodified mirrors of community conversions (credited above) of poolside's official Laguna-S-2.1-DFlash ; poolside does not endorse this redistribution.