dmusingu / cxr-vitl14-multiobjective-regsteps

CXR ViT-L/14 — multiobjective regsteps

ViT-L/14 chest-radiograph vision encoder, pretrained from scratch with the objective: Joint contrastive+captioning, region-phrase loader drives the epoch; eff. neg batch 384.

Part of a controlled study comparing pretraining objectives for chest-X-ray vision encoders (all share the same ViT-L/14 backbone, data, and budget).

Files

Architecture & training

• ViT-L/14, 384x384 input -> 729 patch tokens (27x27 grid), trained from scratch in bf16. Text side (where present) uses the GPT-2 tokenizer (vocab 50257).
• Training data: MIMIC-CXR (reports) and Chest ImaGenome (region-phrase / anatomy boxes).
• Objective: Joint contrastive+captioning, region-phrase loader drives the epoch; eff. neg batch 384.

Downstream results (frozen-feature transfer)

STRONGEST all-rounder: #1 PG-box (0.275), AD (0.073), DiffVQA (0.660); tied-#1 VQA (0.285).

Full per-task comparison across all 9 from-scratch encoders is in the project logbook.

Usage (vision encoder)

import torch
ckpt = torch.load("encoder_final.pt", map_location="cpu", weights_only=False)
vision_state = ckpt["encoder_state"]   # 294 tensors, ViT-L/14
# load into your ViT-L/14 implementation, then forward 384x384 images -> [B, 729, 1024]

Intended use & limitations

• Research use only. Frozen feature extraction / fine-tuning for chest-X-ray tasks.
• NOT a diagnostic device. No clinical or patient-facing use.
• Trained only on adult frontal/lateral CXR distributions of the source datasets; may not generalize to other modalities, body regions, populations, or acquisition settings.

⚠️ License / Data Use Agreement

These weights are derived from MIMIC-CXR and Chest ImaGenome , which are distributed under the PhysioNet Credentialed Health Data Use Agreement . Redistribution and use of models derived from these data are subject to that DUA — you must hold the appropriate credentialed access and comply with its terms. Do not make this repository public or share access without confirming your DUA permits sharing derived model weights.

Citation

If you use this encoder, please cite the source datasets (MIMIC-CXR; Chest ImaGenome) and this project.