ThanhLe0125 / ebd-math

E5-Math-Vietnamese: MRR-Optimized Exact Chunk Retrieval

Model Overview

Fine-tuned E5-base model optimized with MRR (Mean Reciprocal Rank) for exact chunk retrieval in Vietnamese mathematics:

• 🎯 Goal : Push correct chunks to rank #1 or #2
• 📊 Hierarchy : Correct > Related > Irrelevant scoring
• ⚡ Efficient : High MRR = fewer chunks needed at inference

Performance Metrics

Training Results

• Best Validation MRR : 0.8555555555555555 (avg rank: 1.168831168831169)
• Training Epochs : 9
• Hierarchy Maintained : N/A

Final Test Results (Used Only Once)

• Test MRR : N/A (avg rank: N/A)
• Test Recall@3 : N/A
• Test Recall@5 : N/A
• Hierarchy Rate : N/A
• Performance Level : N/A

MRR Advantage

MRR = Mean of (1/rank_of_correct_chunk)

Examples:
- Correct chunk at rank #1 → MRR contribution = 1.0 
- Correct chunk at rank #2 → MRR contribution = 0.5
- Correct chunk at rank #5 → MRR contribution = 0.2
- Correct chunk at rank #10 → MRR contribution = 0.1

High MRR → Correct chunks consistently at top positions!

Usage

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

# Load MRR-optimized model
model = SentenceTransformer('ThanhLe0125/ebd-math')

# ⚠️ CRITICAL: Must use E5 prefixes
query = "query: Định nghĩa hàm số đồng biến là gì?"
chunks = [
    "passage: Hàm số đồng biến trên khoảng (a;b) là...",  # CORRECT
    "passage: Ví dụ bài tập về hàm đồng biến...",        # RELATED
    "passage: Phương trình bậc hai có dạng..."           # IRRELEVANT
]

# Get rankings (MRR-optimized for top positions)
query_emb = model.encode([query])
chunk_embs = model.encode(chunks)
similarities = cosine_similarity(query_emb, chunk_embs)[0]

# With high MRR, correct chunk should be at rank #1 or #2
ranked_indices = similarities.argsort()[::-1]
for rank, idx in enumerate(ranked_indices, 1):
    print(f"Rank {rank}: Score {similarities[idx]:.4f} - {chunks[idx][:50]}...")

# Expected with MRR optimization:
# Rank 1: Correct chunk (score ~0.85)
# Rank 2: Related chunk (score ~0.45) 
# Rank 3: Irrelevant chunk (score ~0.1)

Inference Optimization

With MRR ≥ 0.7, you typically only need to return top 2-3 chunks to find the correct answer:

# Efficient inference - return only top 3
top_3_indices = similarities.argsort()[::-1][:3]
top_chunks = [chunks[i] for i in top_3_indices]

# High probability correct chunk is in top_chunks[0] or top_chunks[1]

Evaluation Methodology

• Training : train_question + val_question with MRR optimization
• Validation : MRR used for early stopping (pushes correct chunks up)
• Test : test_question used ONLY ONCE for final evaluation
• Metrics : MRR (primary), Recall@3/5 (secondary) for exact chunk retrieval

Key Features

• ✅ MRR-optimized : Correct chunks consistently at top positions
• ✅ Inference efficient : Need fewer chunks to find correct answer
• ✅ Hierarchical : Maintains Exact > Related > Irrelevant ranking
• ✅ Vietnamese math specialized : Optimized for educational content
• ✅ Proper ML practices : Clean train/val/test split
• ⚠️ E5 prefixes required : Use 'query:' and 'passage:' prefixes

Perfect For

• Educational Q&A requiring exact answers at top positions
• Efficient retrieval systems (fewer API calls needed)
• Vietnamese mathematics tutoring with quick response
• Content ranking where correct answer must be rank #1 or #2

Performance Interpretation

• MRR ≥ 0.8 : 🌟 Excellent (correct chunks avg rank ≤ 1.25)
• MRR ≥ 0.7 : 🎯 Very Good (correct chunks avg rank ≤ 1.4)
• MRR ≥ 0.5 : 👍 Good (correct chunks avg rank ≤ 2.0)
• MRR < 0.5 : 📈 Needs improvement

Trained on 29/06/2025 with MRR optimization for exact chunk retrieval.