dmusingu / cxr-vitl14-lgc

CXR ViT-L/14 — lgc

ViT-L/14 chest-radiograph vision encoder, pretrained from scratch with the objective: Local generative (GCAP/AREF/CAREF box<->phrase) + global image<->report contrastive; the fourth cell of the global/local x generative/contrastive grid.

Part of a controlled study comparing pretraining objectives for chest-X-ray vision encoders (all share the same ViT-L/14 backbone, data, and budget).

Files

Architecture & training

• ViT-L/14, 384x384 input -> 729 patch tokens (27x27 grid), trained from scratch in bf16. Text side (where present) uses the GPT-2 tokenizer (vocab 50257).
• Training data: MIMIC-CXR (reports) and Chest ImaGenome (region-phrase / anatomy boxes).
• Objective: Local generative (GCAP/AREF/CAREF box<->phrase) + global image<->report contrastive; the fourth cell of the global/local x generative/contrastive grid.

Downstream results (frozen-feature transfer)

PRELIMINARY, not comparable to the others: trained 1 epoch on a single A40 (vs 8x A100 for the rest), then stopped early. Local head strong already (val grounding IoU 0.781, [email protected] 0.895); global contrastive head at chance (R@1 0.063) because the single-GPU negative pool is 24 vs 384. encoder_final.pt = the epoch-1 best checkpoint (extracted from model_best.pt).

Full per-task comparison across all 9 from-scratch encoders is in the project logbook.

Usage (vision encoder)

import torch
ckpt = torch.load("encoder_final.pt", map_location="cpu", weights_only=False)
vision_state = ckpt["encoder_state"]   # 294 tensors, ViT-L/14
# load into your ViT-L/14 implementation, then forward 384x384 images -> [B, 729, 1024]

Intended use & limitations

• Research use only. Frozen feature extraction / fine-tuning for chest-X-ray tasks.
• NOT a diagnostic device. No clinical or patient-facing use.
• Trained only on adult frontal/lateral CXR distributions of the source datasets; may not generalize to other modalities, body regions, populations, or acquisition settings.

⚠️ License / Data Use Agreement

These weights are derived from MIMIC-CXR and Chest ImaGenome , which are distributed under the PhysioNet Credentialed Health Data Use Agreement . Redistribution and use of models derived from these data are subject to that DUA — you must hold the appropriate credentialed access and comply with its terms. Do not make this repository public or share access without confirming your DUA permits sharing derived model weights.

Citation

If you use this encoder, please cite the source datasets (MIMIC-CXR; Chest ImaGenome) and this project.