ekacare / parrotlet-v-2.5-pro

parrotlet-v-2.5-pro

A fine-tune of google/gemma-4-E2B-it (Gemma 4, E2B — ~2B effective parameters) for structured, schema-driven extraction from Indian medical records: prescriptions, invoices, lab reports, and discharge summaries, plus PII detection.

The language-model backbone was fully fine-tuned; the vision and audio towers and the multimodal projector were kept frozen. Weights are released in uniform bf16 , matching the model's declared dtype and training precision.

How to use

The model is prompted with a document image plus one of the tags below as the user turn. No other instruction text is required — the tag alone selects the extraction schema.

pip install torch torchvision "transformers>=5.14" accelerate pillow

# The repo is gated — authenticate first (or set HF_TOKEN)
hf auth login

Keep the transformers>=5.14 floor: versions below 5.6 either lack the gemma4 architecture entirely or (5.5.0) load it with silently re-initialized layers. Expect a ~11 GB download and ~10.7 GiB peak VRAM in bf16.

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image

model_id = "ekacare/parrotlet-v-2.5-pro"
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

image = Image.open("prescription.jpg")

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": "<prescription>"},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True,
    tokenize=True, return_dict=True, return_tensors="pt",
).to(model.device)

output = model.generate(**inputs, max_new_tokens=2048, do_sample=False)
result = processor.decode(output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
print(result)

Supported tags

Validate the tag string before sending: a mistyped tag does not error — the model returns valid-looking but empty JSON (e.g. <lab_repot> yields {"lab_report_elements": {"tests": []}} ).

Evaluation

Internal evaluation on a held-out set of 1,147 pages across five document types, judged by an LLM rubric judge. Overall rubric score = mean(coverage, grounding) , pooled from per-page judge scores. Baselines shown are this model's own un-fine-tuned base and four frontier models evaluated on the same pages with the same rubric.

Blended score, all five tasks

Fine-tuning improves the blended score by ~40 points over the base model and clears every frontier model tested by 1.7–4.4 points. Gemini 3.6 Flash is the closest. The lead is entirely in grounding — this model's coverage (82.35) is third-best, behind Gemini and Claude Sonnet 5, while its grounding of 89.09 is 4 points above the best frontier model. It reproduces slightly less of the reference, but far less of what it writes is unsupported.

Per document type (overall rubric score)

Wins 4 of 5 document types. Invoice is the exception and the model's weakest task — Gemini 3.6 Flash leads there, just ahead of Claude Sonnet 5. PII is the widest margin, ~12.5–16 points clear of every frontier model.

Lab reports: complex vs. simple

Complex = report-style/non-tabular modalities (echo, ECG, manometry, histopathology); simple = standard tabular reports.

The complex cohort is where the gap closes: Gemini 3.6 Flash comes within 0.12 points of this model there, while still trailing by 3.7 on simple pages. Gemini also has the smallest complex→simple drop of any model tested (8.0 points, against this model's 11.6), so its advantage is relative robustness to non-tabular layouts rather than better extraction overall.

Limitations

• Weakest on invoices relative to frontier models; this is the one document type where a fine-tune does not lead. Handwritten bills are the hardest invoices for every model tested — all of them score ~30 points lower on handwritten item rows than on printed ones — and this model's characteristic failure there is giving up entirely rather than extracting partially: it returns an empty line-item list on 18 of 200 invoices, 16 of which have hand-written items. Recovering those pages would lift the invoice score from 74.41 to ~76.2, closing about a fifth of the gap; the remainder is spread across ordinary printed invoices.
• Complex/non-tabular lab report layouts (echo, ECG, manometry, histopathology) show the largest remaining headroom, and are the one cohort where a frontier model (Gemini 3.6 Flash) draws level.
• Evaluated only on an internal held-out set; not benchmarked on out-of-distribution document layouts, languages, or non-Indian medical record formats.
• This is a document-parsing model, not a diagnostic or clinical decision-making tool. Outputs (including PII detection) should be reviewed by a human before use in any compliance-sensitive workflow.

License

Apache-2.0. The base model, google/gemma-4-E2B-it , is released by Google under Apache 2.0 (unlike Gemma 1–3's Gemma Terms of Use), so this fine-tune is distributed under Apache-2.0 as well. Google publishes a Prohibited Use Policy and Intended Use statement for the Gemma 4 family that users should review.

Citation

If you use this model, please cite:

@software{parrotlet_v_2_5_pro,
  author = {{Eka Care}},
  title  = {Parrotlet-v 2.5 Pro: schema-driven extraction from Indian medical documents},
  year   = {2026},
  url    = {https://huggingface.co/ekacare/parrotlet-v-2.5-pro}
}