dmusingu / lapvqa-diffvqa

LAPVQA — Differential VQA (Frozen Off-the-shelf Encoders)

Part of the LAPVQA collection .

Description

Task heads for Differential VQA : given a prior and a current chest X-ray, answer questions about radiological changes. Trained on MIMIC-Diff-VQA with five frozen encoders. Each .pt file is a plain state dict of DiffVQAHead .

Architecture — DiffVQAHead

vis_proj   : Linear(vis_dim → 512)   # shared for both images
frame_emb  : Embedding(2, 512)       # 0=reference, 1=current
memory     : [ref_proj + frame_emb(0) ; curr_proj + frame_emb(1)]  → [B, 2N, 512]
tok_emb    : Embedding(50257, 512)
pos_emb    : Embedding(200, 512)
decoder    : 6 × TransformerDecoderLayer (pre-norm)
lm_head    : Linear(512 → 50257, bias=False)

Results (test set)

Loading

import torch
import tiktoken
from lapvqa.diffvqa.model import DiffVQAHead

ckpt = torch.load("coca_best.pt", map_location="cpu")
head = DiffVQAHead(vis_dim=768)   # adjust vis_dim per encoder
head.load_state_dict(ckpt)
head.eval()

enc = tiktoken.get_encoding("gpt2")
bos_id = eos_id = enc.eot_token

# curr_vis, ref_vis: [B, N, vis_dim] — patch tokens from the frozen encoder
answers = head.generate(
    curr_vis    = curr_vis,
    ref_vis     = ref_vis,
    prompt_ids  = question_ids,   # [B, Q]
    bos_id      = bos_id,
    eos_id      = eos_id,
    max_new_tokens = 128,
)
decoded = [enc.decode(ids) for ids in answers]