AbstractPhil / geolip-clip-vit-large-patch14-ctx576

GEOLIP-CLIP-ViT-L/14 ctx576 — Memory-Extended CLIP Text Encoder

Extends CLIP-ViT-L/14 text encoder from 77 tokens to 576 effective context via geometric memory bank, trained by distillation from ModernBERT-large.

Key Results

Usage

from transformers import AutoModel

model = AutoModel.from_pretrained(
    "AbstractPhil/geolip-clip-vit-large-patch14-ctx576",
    trust_remote_code=True)
model = model.to("cuda").eval()

# Single text → 768-dim CLIP-compatible embedding
embedding = model.encode("A vast sweeping landscape of rolling green hills...")
print(embedding.shape)  # (768,)

# Batch
embeddings = model.encode([
    "A cat sleeping on a warm blanket",
    "A medieval castle overlooking a turbulent sea with waves crashing "
    "against ancient rocks and seabirds wheeling overhead against a sky "
    "painted in shades of grey and gold as the sun breaks through clouds"
])
print(embeddings.shape)  # (2, 768)

# HuggingFace forward() API
output = model(texts=["A photo of a dog"])
print(output.last_hidden_state.shape)  # (1, 1, 768)

Architecture

Long caption (576 tokens)
    │
    ├── ModernBERT-large ──── 4096 ctx, 1024-dim ──── teacher (training only)
    │   (frozen, 395M)
    │
    └── CLIP-L text enc ──── 18 tok × 32 segments ── student
        (frozen, 768-dim, 123M)
        + Geometric Memory System (34.5M trainable)
            ├── Depth compressor (6-layer profile → 768-dim anchor)
            ├── Memory bank (64 anchors, 2-layer cross-attention)
            ├── CLIP cross-attention (memory → CLIP hidden states)
            ├── GRU gate (memory state updates)
            ├── Layer fusion (learned weighted sum of 6 CLIP layers)
            └── Teacher projector (768 → 1024, Procrustes-initialized)

How It Works

1. Long text split into 18-token segments (stride 14, overlap 4)
2. Each segment: frozen CLIP text encoder forward (standard, untouched)
3. Memory tokens cross-attend to CLIP's multi-layer hidden states
4. Depth-profile anchors stored in geometric memory bank
5. GRU gate controls rolling memory state across segments
6. Final output: 768-dim embedding in CLIP's text space

Training

• Teacher: ModernBERT-large (frozen, 4096 context)
• Losses: InfoNCE(student→teacher) × 1.0 + SVD Procrustes regularizer × 0.3 + pentachoron CV × 0.05
• Data: CC12M with LLaVA-next detailed captions (50K train, 2K val, mean 96 tokens)
• Hardware: NVIDIA RTX PRO 6000 Blackwell (102 GB VRAM)
• Training time: ~103 minutes (10 epochs × ~10.3 min/epoch)
• Batch size: 64

Geometric Properties

• Procrustes init: CLIP→ModernBERT cos 0.001 → 0.816 (strongest alignment seen across all GEOLIP experiments)
• CV trajectory: 0.185 (E1) → 0.162 (E10), stable in the 0.16-0.19 band
• CV regularization on bank anchors prevented projector shortcut collapse
• 18-token segments forced genuine memory accumulation (vs 55-token segments which plateaued at 0.670)

Why 18-Token Segments?

55-token segments → 2 segments per caption → pentachoron CV dead (needs 5+ anchors) → projector shortcut → plateau at m_acc=0.670.

18-token segments → 7 segments per caption → CV active → memory bank forced to accumulate → m_acc=0.945.

The geometry regularization must be applied at the bottleneck (bank anchors). If it can't activate, the model finds cheap projector shortcuts.

Training Metrics

Full training metrics available in metrics.json . TensorBoard logs in tensorboard/ .

Training Curve

*E2 val dip: cosine LR transient at epoch boundary, recovered by E3.

GEOLIP Family

All use the same blueprint: frozen expert teacher + Procrustes initialization + InfoNCE alignment force + pentachoron CV on the bottleneck representation.

License

Apache 2.0