ajvikram / cranebear-vlm-4b-gguf

CraneBear VLM 4B (GGUF) — transcript + optional photo → extraction JSON

A LoRA fine-tune of Qwen3-VL-4B-Instruct that turns a panic emergency voice transcript, with an optional photo, into the same 7-key extraction JSON the CraneBear text model emits, for a human dispatcher. Ships as a Q4_K_M text GGUF plus an f16 multimodal projector, for llama.cpp / llama-server / the iOS mtmd path.

The model extracts. A human decides. It never triages, dispatches, or advises. Severity bands are evidence labels for a trained dispatcher.

This repo replaces ajvikram/emergency-vision-triage-2b-gguf , which was a text-only decoder with no vision tower despite its name (documented in the training write-up); that repo is deprecated.

Two checkpoints

For reference, the partner's production text model ( cranebear-extraction-4b , round 3) scores 91.7% / 0% on the same set, on-contract. Both checkpoints here were trained on the same 800-row round-3 text set; r3 adds 1,018 (transcript, photo) pairs.

Status: what is and is not established

Established, measured:

• Text contract: both checkpoints pass the partner's blocking gate (0% missed-IMMEDIATE, asserted) and the ≥90% macro gate on their 24-case set.
• The vision path is real: r3 reads hazards from photos (7/7 on the photo-adds slice of a 60-case photo set), ignores irrelevant photos in 14/18 cases, and over-triages far less than a text-only checkpoint shown the same photos (false-IMMEDIATE 3/16 vs 16/31).
• Served through llama-server and through the partner's Swift/ mtmd path, the two stacks agree on these numbers.

Not established — do not treat as production-safe for photo input yet:

• The 60-case photo eval set carries proposed, unratified labels and only 34 sourced photos. On it, r3 still has 1 missed-IMMEDIATE, 1 photo-subtracted severity, 2 photo-only HIGH confidences.
• Those last two classes go to zero for every checkpoint when the caller applies the text-first merge (below). That merge is a caller-side guardrail; it is not inside the weights.
• The photo training taught four rules (a photo may add a hazard; a photo never subtracts; photo-only evidence never HIGH; an irrelevant photo changes nothing). It did not teach a photo supplying a rule-1 sign (a person under rubble) or a casualty count — no public image set covers that ethically.
• Never run on a phone. All numbers are llama.cpp on a Mac.

Serving contract (must match exactly)

• Prompt: the SYSTEM text in Modelfile.vlm-v2-r3 (byte-identical to the CraneBear text model's round-3 prompt). With a photo, append the paragraph in photo_addendum.py to the system prompt and place the image before Transcript: in the user turn.
• No think block. Qwen3-VL-Instruct's template has none; rendering one is a served/trained mismatch.
• Grammar: cranebear-extraction-presence.gbnf . The model emits a presence-encoded object (no nulls); the consumer maps sentinels to nulls.
• Greedy (temperature 0). Photos downscaled to 512 px longest side (~200 image tokens).

llama-server -m vlm-v2-r3-q4_k_m.gguf --mmproj mmproj-vlm-v2-r3-f16.gguf -c 4096 -ngl 99
# POST /v1/chat/completions with the system prompt (+ addendum), an image_url
# content part, "Transcript:\n<text>", and "grammar": <the gbnf>.

Recommended caller-side merge for photo input (what the partner's Guardrails.mergePhoto does): run the text-only pass first, then the photo pass; take severity = max on MINOR < DELAYED < IMMEDIATE (DECEASED only from text), OR the hazard bits, keep a stated casualty count, and cap photo-only severity confidence at MEDIUM. Measured: this removes every "photo subtracted" and "photo-only HIGH" case for every checkpoint tested.

Training

• Base Qwen/Qwen3-VL-4B-Instruct (16-bit), LoRA r=16 on the language tower, vision encoder frozen, early stopping on eval loss, best checkpoint loaded.
• r2: 800 text rows (partner-schema, presence-encoded), 4 epochs, ~3.4 h.
• r3: + 1,018 photo pairs from 880 Wikimedia Commons photos (CC0 / CC BY / CC BY-SA / PD, recorded per file) with deterministic labels derived from each photo's category and the four photo rules — no teacher model. 300 steps, ~6.9 h. mmproj is byte-identical to the base's (frozen tower).
• Hardware: NVIDIA DGX Spark (GB10). Reproduction scripts, data engines, eval harnesses and the full round-by-round record (including a documented negative round 4) live in the slm-factory repo.

Files

• vlm-v2-r2-q4_k_m.gguf , vlm-v2-r3-q4_k_m.gguf — text models (Q4_K_M, 2.5 GB)
• mmproj-vlm-v2-r3-f16.gguf — vision projector for r3 (839 MB)
• cranebear-extraction-presence.gbnf — grammar
• Modelfile.vlm-v2-r2 , Modelfile.vlm-v2-r3 — Ollama Modelfiles (SYSTEM block = the prompt)
• photo_addendum.py — the photo paragraph, with its SHA
• evals/ — the raw eval printouts behind every number above
• SHA256SUMS