tencent / EVIE-4.5B

---

📢 Release Announcement : All model weights, training pipelines, token compression algorithms (HAC), and evaluation suites have been fully open-sourced in this repository. Full technical details, architectural ablations, and the formal research paper will be updated in an upcoming release.

---

🌟 Highlights

• Top-Tier Benchmark Performance : 66.75 on ViDoRe V3 for EVIE-8B and 66.02 for EVIE-4.5B with single-projection Prefix-MRL.
• ⚡ Prefix-MRL Elasticity : Single 2048D linear projection. Freely truncate at runtime into ${64, 128, 256, 512, 1024, 2048}$ dimensions without separate models.
• 📦 Ultra-Compact Index (HAC) : Training-free Hierarchical Agglomerative Clustering compresses token counts from ~750 down to 32 vectors/page , slashing index storage to 3.81 GiB per million pages .
• 🌐 138 Multilingual Tasks Evaluated : Thoroughly evaluated across ViDoRe V1, V2, V3, and JinaVDR across 4 metric families (nDCG, Recall, MAP, MRR @1/5/10).
• 🔬 ARD Distillation Recipe : Anchor-preserving, capacity-aware relation distillation reproducing full student training from the 8B teacher.

---

🧠 Architecture & Technical Highlights

 Query Text  ────────► ColQwen3.5 (BiDir Attention) ────► Elastic Multi-Vectors (64D–2048D)
                                                                    │
                                                           MaxSim Matching
                                                                    │
 Doc Image   ────────► ColQwen3.5 (Vision Encoder)  ────► HAC Compression ──► 32 Vectors / Page

• Late-Interaction Multi-Vector Paradigm : Unlike dense single-vector retrieval that collapses high-resolution document pages into a single point, EVIE preserves fine-grained visual details (complex tables, layout structures, charts, and small typography) through token-level representations, scoring relevance via late-interaction MaxSim:

$S(Q,D)=\underset{i=1}{\overset{\mid Q\mid }{\sum }}\underset{j=1}{\overset{\mid D\mid }{\max }}(q_i\cdot d_j)S(Q,\; D)\; =\; \backslash sum\_\{i=1\}^\{|Q|\}\; \backslash max\_\{j=1\}^\{|D|\}\; (q\_i\; \backslash cdot\; d\_j)$ S ( Q , D ) = i = 1 ∑ ∣ Q ∣ ​ j = 1 max ∣ D ∣ ​ ( q i ​ ⋅ d j ​ )

• Prefix-MRL (Single-Head Elastic Representation) : EVIE-4.5B introduces single-projection Prefix-MRL. A single 2048D linear projection natively supports runtime truncation down to {64, 128, 256, 512, 1024, 2048} dimensions without maintaining multiple heads or separate checkpoints.
• ARD (Anchor-preserving Relation Distillation) : The 4.5B student is distilled from the 8B teacher using token-relation topological geometry, hard-negative margin calibration, and anchor-preserving alignment, maintaining peak retrieval accuracy even under low-dimensional prefixes.
• HAC Token Compression (Hierarchical Agglomerative Clustering) : A plug-and-play, training-free token reduction algorithm that aggregates visual patch tokens into 32 or 64 semantic centroids in joint feature-spatial space, reducing 1M-page index footprints to as little as 3.81 GiB .

---

📊 Comprehensive ViDoRe Leaderboard Comparison

Performance comparison across modern multi-vector late-interaction visual document retrievers on ViDoRe:

---

🔍 ViDoRe V3 Per-Domain Breakdown (nDCG@10)

---

🎯 Prefix-MRL Elastic Multi-Vector Head

EVIE-4.5B embeds document and query tokens with a single 2048D linear projection head trained via ARD . You can truncate the channel dimension on-the-fly without maintaining different models:

Full Projection (2048D)  [========================================================] 66.02
Prefix 1024D             [============================]                             65.94
Prefix 512D              [==============]                                           65.90
Prefix 256D              [=======]                                                  65.68
Prefix 128D              [===]                                                      65.27
Prefix 64D               [=]                                                        64.51

---

🗜️ Token Compression (HAC)

Raw late-interaction representations keep all visual patch vectors (~750 vectors/page), requiring substantial storage. EVIE integrates Hierarchical Agglomerative Clustering (HAC) in a joint semantic-position space to cluster page tokens at indexing time:

$z=\text{L2}\left[\right(1-w\left)\text{L2}\right(v)+wp],{\mu }_c=\text{L2}\left({\text{mean}}_{i\in c}{v}_i\right)z\; =\; \backslash text\{L2\}\backslash left[(1-w)\backslash text\{L2\}(v)\; +\; w\; p\backslash right],\; \backslash quad\; \backslash mu\_c\; =\; \backslash text\{L2\}\backslash left(\backslash text\{mean\}\_\{i\; \backslash in\; c\}\; v\_i\backslash right)$ z = L2 [ ( 1 − w ) L2 ( v ) + wp ] , μ c ​ = L2 ( mean i ∈ c ​ v i ​ )

Production Ready SKUs

---

📋 Complete 138-Task Evaluation Matrix

Protocol paired-all-pages-dedup+process_queries+ndcg2r-20260827 ($\text{MVT} = 1024$, bidirectional attention):

---

⚡ Quick Start

ColPali Engine

Installation

git clone https://github.com/Tencent/EVIE.git
cd EVIE
pip install -r requirements.txt
export PYTHONPATH="$(pwd)/colpali${PYTHONPATH:+:$PYTHONPATH}"

Self-Contained Python Inference

import torch
from PIL import Image
from colpali_engine.models import ColQwen3_5, ColQwen3_5Processor
from colpali_engine.models.qwen3_5.colqwen3_5.modeling_colqwen3_5 import set_active_head

model_id = "tencent/EVIE-4.5B"

# 1. Load model with FlashAttention and bidirectional attention
model = ColQwen3_5.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="cuda",
    attn_implementation="flash_attention_2",
).eval()
model.enable_bidirectional_attention()

# 2. Select any Matryoshka head: 64, 128, 256, 512, 1024, or 2048
set_active_head(model, 128)

# 3. Process query and document image
processor = ColQwen3_5Processor.from_pretrained(model_id)
images = [Image.open("examples/demo/pages/q3_revenue.png").convert("RGB")]
queries = ["What is the total quarterly revenue?"]

image_batch = processor.process_images(images).to(model.device)
query_batch = processor.process_queries(queries).to(model.device)

# 4. Generate multi-vector representations and late-interaction score
with torch.inference_mode():
    image_embeddings = model(**image_batch)
    model.rope_deltas = None  # Reset RoPE deltas before text query forward
    query_embeddings = model(**query_batch)

scores = processor.score(query_embeddings, image_embeddings)
print("Late-interaction MaxSim Relevance Score:", scores)

Sentence Transformers

pip install "sentence-transformers[image] @ git+https://github.com/huggingface/sentence-transformers.git"

from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("tencent/EVIE-4.5B")

queries = [
    "What is the variable represented on the y-axis of the graph?",
    "Total outlay is maximum in which year?",
]
documents = [
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc1.jpg",
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc2.jpg",
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc3.jpg",
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc4.jpg",
]

query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# torch.Size([23, 2048]) torch.Size([991, 2048])

scores = model.similarity(query_embeddings, document_embeddings)
print(scores)
# tensor([[17.2500,  7.6084,  6.7920,  4.8398],
#         [ 4.5312, 12.7812,  4.3530,  4.2368]])

Documents can be URLs, local image paths, or PIL.Image objects. The example returns the full 2048-dimensional token embeddings. Scores can vary slightly with dtype and attention backend. To use a smaller Prefix-MRL dimension, slice and renormalize:

from torch.nn.functional import normalize

dimension = 128
query_embeddings = [normalize(embedding[..., :dimension], dim=-1) for embedding in query_embeddings]
document_embeddings = [normalize(embedding[..., :dimension], dim=-1) for embedding in document_embeddings]
scores = model.similarity(query_embeddings, document_embeddings)

---

📂 Repository Layout

EVIE/
├── model.safetensors         # EVIE-4.5B weights (safetensors)
├── config.json               # Model configuration (Prefix-MRL, max 2048)
├── processor_config.json     # Multimodal processor config
├── infer.py                  # Standalone inference & scoring CLI
├── colpali/                  # ColQwen3.5 + Prefix-MRL + ARD
├── code/
│   ├── teacher/              # EVIE-8B training arms and soup merging
│   ├── student/              # EVIE-4.5B Prefix-MRL / ARD distillation
│   ├── shared/               # Data loaders, adapter merges, and 138-task eval harness
│   └── compress/             # Training-free HAC token compression pipeline
├── examples/demo/            # 8-page retrieval demo (run.sh)
└── env.sh.example            # Environment variables template

---

🔬 Training & Distillation Reproduction

Student training executes Prefix-MRL plus ARD: capacity-aware relation distillation and margin distillation against frozen EVIE-8B , with a Preview-anchor term on the 128D prefix:

cp env.sh.example env.sh
source env.sh

# Step 1: Train EVIE-8B teacher arms and merge
bash code/teacher/run.sh

# Step 2: Distill EVIE-4.5B student with ARD loss
export TEACHER_DIR=../Evie-8B
bash code/student/run.sh

# Step 3: Run comprehensive 138-task evaluation
MODEL_DIR=. RUN_NAME=evie-4.5b bash code/shared/eval_run.sh

---

📚 Citation

@misc{tencent2026evie,
  title        = {EVIE: High-Performance Multilingual Visual Document Retrieval with Matryoshka Embeddings and Token Compression},
  author       = {Wang, Zifei and Wen, Wei},
  year         = {2026},
  howpublished = {\url{https://github.com/Tencent/EVIE}},
  note         = {Corresponding author: Wei Wen <[email protected]>}
}

---

📄 License

This repository is licensed under the Apache-2.0 License .

---

👥 Authors