ekacare / med-doc-classifier

Eka Doc Classifier (~96M)

A compact vision model for classifying health-app uploads — prescriptions shot at an angle, X-rays, Aadhaar cards, discharge summaries, the occasional accidental selfie. A frozen SigLIP2-base tower ( google/siglip2-base-patch16-naflex ) with lightweight heads answers three questions about an image in a single pass:

• What it is — a two-level label: 5 groups (L1) → 27 leaf classes (L2)
• Is it medical? — binary head
• Is it handwritten? — binary head

The model was built by training many specialised models, ensembling them, and distilling the ensemble into this one compact student.

Results

Full internal test = 16,204 images (see benchmark_results.json ).

int8 is effectively lossless; int4 trades about a point of accuracy for a model over 3× smaller that runs comfortably on a plain CPU. It matches or beats models several times its size, including a 428M SigLIP2-SO400M variant (96.24%) and a 256M generative VLM baseline (90.42%).

Taxonomy (5 L1 groups → 27 L2 leaves)

Document  (10 leaves)
    ├─ Letter                 (key: letter)
    ├─ Other document         (key: other_document)
    ├─ Scan interpretation    (key: scan_interpretation)
    ├─ Lab / diagnostic report (key: lab_diagnostic_report)
    ├─ OPD Consultation Record (key: opd_consultation_record)
    ├─ Discharge summary      (key: discharge_summary)
    ├─ Insurance document     (key: insurance_document)
    ├─ Form                   (key: form)
    ├─ Invoice / bill         (key: invoice_bill)
    ├─ Certificate            (key: certificate)
Card / credential  (7 leaves)
    ├─ ABHA card              (key: abha_card)
    ├─ Aadhaar card           (key: aadhaar_card)
    ├─ PAN card               (key: pan_card)
    ├─ Voter ID               (key: voter_id)
    ├─ PMJAY / Ayushman       (key: pmjay_ayushman)
    ├─ Insurance e-card       (key: insurance_e_card)
    ├─ Other card             (key: other_card)
Diagnostic imaging  (5 leaves)
    ├─ X-ray                  (key: x_ray)
    ├─ CT                     (key: ct)
    ├─ MRI                    (key: mri)
    ├─ Ultrasound             (key: ultrasound)
    ├─ Other (Diagnostic imaging) (key: other_diagnostic_imaging)
Body images  (3 leaves)
    ├─ Skin / wound           (key: skin_wound)
    ├─ Headshot               (key: headshot)
    ├─ Other (Body images)    (key: other_body_images)
Miscellaneous  (2 leaves)
    ├─ Medication image       (key: medication_image)
    ├─ Other (Miscellaneous)  (key: other_miscellaneous)

Authentication

The repo is gated — authenticate before loading:

hf auth login                    # interactive, stores the token locally
# or non-interactive (CI, servers):
export HF_TOKEN="your-access-token"

Generate a token under Hugging Face → Settings → Access Tokens.

Usage

from transformers import AutoModel
from PIL import Image

model = AutoModel.from_pretrained("ekacare/med-doc-classifier", trust_remote_code=True).eval()
img = Image.open("doc.jpg")

model.classify(img)

// example return value
{
  "l2":          { "source": "flat", "key": "lab_diagnostic_report",
                    "value": "Lab / diagnostic report", "confidence": 0.97 },
  "l1":          { "value": "Document", "source": "inferred_from_flat_l2" },
  "medical":     { "value": "medical", "confidence": 0.99, "p_positive": 0.99 },
  "handwritten": { "value": "printed", "confidence": 0.98, "p_positive": 0.02 }
}

By default L2 comes from a flat 27-way head and L1 is inferred from it. Other options:

model.classify(img, scope="l1")           # L1 group only (no L2)
model.classify(img, scope="hierarchical") # L1 head → that group's leaf L2 head
model.classify(img, l1="Document")        # fix L1 → that group's leaf L2 head
model.classify(img, top_k=3)              # top-3 candidates for the multi-class heads

medical and handwritten are independent binary heads — disable them with medical=False / handwritten=False .

Quantized loading (optimum-quanto — CPU & GPU)

Only the vision tower is quantized; the heads stay full precision.

# pip install optimum-quanto
from transformers.dynamic_module_utils import get_class_from_dynamic_module

load_classifier = get_class_from_dynamic_module(
    "modeling_siglip2_hier.load_classifier", "ekacare/med-doc-classifier")

model = load_classifier("ekacare/med-doc-classifier", quantization="int4")  # None | "int8" | "int4"
model.classify(img)

Working from a clone instead? from modeling_siglip2_hier import load_classifier inside the repo directory works too, and load_classifier(".", quantization="int8") loads straight from the checkout.

Citation

If you use this model, please cite:

@software{med_doc_classifier,
  author = {{Eka Care}},
  title  = {Med Doc Classifier: hierarchical classification of health-app document uploads},
  year   = {2026},
  url    = {https://huggingface.co/ekacare/med-doc-classifier}
}