dmusingu / lapvqa-pretrain-captioning

LAPVQA — Pretrain (Captioning)

Part of the LAPVQA collection .

Description

A ViT-L/14 encoder + 6-layer causal decoder trained from scratch on MIMIC-CXR to generate full radiology reports from chest X-ray images. Unlike the contrastive pretrain variants, the generative objective forces the encoder to retain fine-grained spatial information sufficient for region-level text generation. The encoder weights ( encoder_final.pt ) serve as the strongest feature extractor in the LAPVQA downstream tasks.

Architecture

Downstream Evaluation (frozen encoder + linear probe)

The captioning-pretrained encoder matches or exceeds the contrastive variants on both classification benchmarks, and is the best-performing encoder on DiffVQA when used downstream.

Files

Usage

import torch
from lapvqa.pretrain.model import CaptioningModel

ckpt = torch.load("model_best.pt", map_location="cpu")
model = CaptioningModel()
model.load_state_dict(ckpt)
model.eval()

# To use only the encoder as a feature extractor:
enc_weights = torch.load("encoder_final.pt", map_location="cpu")
model.vision_encoder.load_state_dict(enc_weights)
# vis_tokens = model.vision_encoder(images)  # [B, 256, 1024]

Citation

If you use these weights please cite MIMIC-CXR:

@article{johnson2019mimic,
  title   = {MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports},
  author  = {Johnson, Alistair EW and others},
  journal = {Scientific data},
  volume  = {6}, pages = {317}, year = {2019}
}