mlboydaisuke / Qwen3.6-27B-CoreAI

Qwen3.6-27B — Core AI (Apple) bundle

The dense Mac-class companion to Qwen3.6-35B-A3B-CoreAI , converted for Apple's Core AI runtime (iOS/macOS 26+ successor to Core ML). Source: Qwen/Qwen3.6-27B (text decoder).

Where the 35B-A3B is a sparse MoE, the 27B is the same Qwen3.5 hybrid decoder run dense — no experts, no router, just the proven token mixers at scale. 64 layers on a 3:1 interleave of GatedDeltaNet linear-attention mixers and gated full attention :

• full attention: head_dim 256, GQA 24 query / 4 KV heads , partial mRoPE θ=1e7, swish output gate;
• linear (GatedDeltaNet): 48 value heads over 16 key heads (GVA — each k/q head shared across three value heads, vs the 35B's two);
• every FFN a dense MLP(17408) (no MoE); untied 248320-vocab lm_head .

27B parameters, all dense → the entire model is read per token. Unlike the 35B-A3B (≈3B active), there is no sparsity to hide behind: this is a true 27B-class decode — the quality of a large dense model at the memory-bandwidth speed that implies on a Mac.

Use it

▶️ Run it (source) — the ChatDemo runner (GUI + CLI, one app for every chat model in the catalog):

git clone https://github.com/john-rocky/coreai-kit
open coreai-kit/Examples/ChatDemo/ChatDemo.xcodeproj
# → Run, then pick "Qwen3.6-27B (dense)" in the model picker

# agents / headless (macOS):
cd coreai-kit/Examples/ChatDemo
swift run chat-cli --model qwen3.6-27b --prompt "What can you do, offline?"

💻 Build with it — complete; the glue is kit API, copy-paste runs:

import CoreAIKit

let chat = try await ChatSession(catalog: "qwen3.6-27b")
let reply = try await chat.respond(to: prompt)
// reply: the answer, generated fully on-device

The take-home is Examples/ChatDemo/Sources/QuickStart.swift — this exact code as one typed function, no UI; the CLI is an argument shell over it, and the GUI drives the same ChatSession across turns for its transcript. Multi-turn? Hold the ChatSession and call respond(to:) per turn — it keeps the conversation history; streamResponse(to:) yields tokens as they decode.

Integration checklist

• SPM: https://github.com/john-rocky/coreai-kit → product CoreAIKit
• Info.plist: none needed
• Entitlements: none needed (macOS)
• First run downloads the model — 28.0 GB (Mac) — then it loads from the local cache (Application Support; progress via the downloadProgress callback)
• Measure in Release — Debug is ~3× slower on per-token host work

Bundle

gpu-pipelined/qwen3_6_27b_decode_int8hu_block32_sym/ — a ready-to-run Core AI LanguageBundle ( .aimodel + metadata.json + tokenizer), 28 GB , decode-only loop-free for Apple's pipelined GPU engine. int8 linear per-block-32 weights + an absmax int8 untied head ( int8hu --head-sym ).

Measured (macOS 27 beta, M4 Max 128 GB, llm-benchmark , COREAI_CHUNK_THRESHOLD=1 )

Numerics in full — 27B fp32 would need ~111 GB RAM, so the oracle is the checkpoint's native bf16 ; the gate is teacher-forced single-step argmax under an oracle-margin≥0.1 rule. The result is cleaner than the 35B-A3B's: int8 adds zero confident-margin flips over full precision. Both int8hu and an fp16 control score 15/16 vs the bf16 oracle and fail the same position (margin 0.50), where fp16 flips byte-identically to int8 — a bf16-oracle-resolution artifact, not an int8 defect. The only int8-vs-fp16 difference anywhere is one sub-0.1-margin tie.

Speed is bandwidth-bound, as a dense 27B at int8 must be: ~28 GB/token → 15.9 tok/s is ~87 % of the M4 Max memory-bandwidth ceiling. (The 35B-A3B decodes faster than this despite more parameters, because only ~3B are active per token — that is the MoE's whole point.)

int4 is a size/speed option, not the quality ship. A linear int4 bundle ( int4lin , ~14 GB, ~2× decode) gates 15/16 too but pays a real cost: it flips a high-confidence position that fp16 and int8 both get right, and its per-position cosine is systematically lower. A mixed-precision middle ground (MLP int4 / attention·GatedDeltaNet·head int8) was tested and rejected — keeping the mixers at int8 repairs int4's flip (confirming the attention/GDN path, not the FFN, is the 4-bit-sensitive part), but the int4 MLP then introduces its own confident flip that edge-layer int8 cannot fix. So there is no quality-preserving speedup between int8 (clean, 15.9 tok/s) and int4 (borderline, ~30): int8 is the quality ship , int4 the size/speed option, nothing useful between.

Mac-only: at 28 GB this is a 64/128 GB-Mac model (far past the iPhone memory limit).

How to run

This is a Core AI bundle for Apple's pipelined LLM engine ( llm-benchmark / llm-runner from apple/coreai-models , plus the community pipelined extra-states patch). The conversion recipe and the full write-up live in the community zoo: github.com/john-rocky/coreai-model-zoo ( zoo/qwen3.6-27b.md ). The decoder reuses the shared qwen3_5.py overlay directly — no MoE files.

COREAI_CHUNK_THRESHOLD=1 llm-benchmark \
    --model gpu-pipelined/qwen3_6_27b_decode_int8hu_block32_sym -p 64 -g 128 -n 3