dmusingu / lapvqa-vqa

LAPVQA — VQA (Frozen Off-the-shelf Encoders)

Part of the LAPVQA collection .

Description

Lightweight task heads for Visual Question Answering on MIMIC-Diff-VQA, trained on top of five frozen off-the-shelf vision encoders. Each .pt file contains only the task head weights; load the encoder separately.

Architecture — VQAHead

vis_proj   : Linear(vis_dim → 512)
tok_emb    : Embedding(50257, 512)   # GPT-2 vocab, weight-tied with lm_head
pos_emb    : Embedding(150, 512)
decoder    : 6 × TransformerDecoderLayer (pre-norm, cross-attn to visual tokens)
lm_head    : Linear(512 → 50257, bias=False)

Results (test set, overall)

Loading

import torch
import tiktoken
from lapvqa.vqa.model import VQAHead

# checkpoint is a plain state dict
ckpt = torch.load("clip-vit-l14_best.pt", map_location="cpu")
head = VQAHead(vis_dim=1024)
head.load_state_dict(ckpt)
head.eval()

# vis_tokens: [B, N, vis_dim] — patch tokens from the frozen encoder
# prompt_ids: [B, Q]           — tokenised question (GPT-2 tokeniser)
enc = tiktoken.get_encoding("gpt2")
bos_id, eos_id = enc.eot_token, enc.eot_token

answers = head.generate(
    vis_tokens  = vis_tokens,
    prompt_ids  = prompt_ids,
    bos_id      = bos_id,
    eos_id      = eos_id,
    max_new_tokens = 64,
)
decoded = [enc.decode(ids) for ids in answers]