ThanhLe0125 / e5-math-ebd

E5-Math-Vietnamese: MRR-Optimized with Base Model Comparison

Model Overview

Fine-tuned E5-base model optimized with MRR (Mean Reciprocal Rank) for exact chunk retrieval in Vietnamese mathematics. Includes comprehensive comparison with base model.

Performance Comparison

Training vs Test Performance

• Best Validation MRR : 0.8388888888888888 (avg rank: 1.1920529801324504)
• Test MRR : 0.8449820788530465 (avg rank: 1.183457051961824)
• Training Epochs : 7

Fine-tuned vs Base Model Comparison

Detailed Recall@k Comparison

Key Improvements from Fine-tuning

✅ MRR Boost : +0.08566308243727583 improvement in Mean Reciprocal Rank ✅ Ranking Quality : Correct chunks moved up by avg 0.13351250196784314 positions
✅ Hit Rate : Better success rates across all Recall@k metrics ✅ Vietnamese Math : Specialized for Vietnamese mathematical content ✅ Hierarchy : Maintains Correct > Related > Irrelevant scoring

Why MRR Matters for Exact Retrieval

MRR optimization pushes correct chunks to top positions:

Before (Base Model):
Rank 1: Related chunk    (MRR contribution: 0.0)
Rank 2: Irrelevant      (MRR contribution: 0.0)  
Rank 3: CORRECT chunk   (MRR contribution: 0.33)

After (Fine-tuned):
Rank 1: CORRECT chunk   (MRR contribution: 1.0)  ⭐
Rank 2: Related chunk   (MRR contribution: 0.0)
Rank 3: Irrelevant     (MRR contribution: 0.0)

Result: 3x better MRR, users find answers immediately!

Usage

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

# Load MRR-optimized model
model = SentenceTransformer('ThanhLe0125/e5-math-ebd')

# ⚠️ CRITICAL: Must use E5 prefixes
query = "query: Định nghĩa hàm số đồng biến là gì?"
chunks = [
    "passage: Hàm số đồng biến trên khoảng (a;b) là...",  # CORRECT
    "passage: Ví dụ bài tập về hàm đồng biến...",        # RELATED
    "passage: Phương trình bậc hai có dạng..."           # IRRELEVANT
]

# Get MRR-optimized rankings
query_emb = model.encode([query])
chunk_embs = model.encode(chunks)
similarities = cosine_similarity(query_emb, chunk_embs)[0]

# With fine-tuning, correct chunk should be at rank #1
ranked_indices = similarities.argsort()[::-1]
print(f"Rank 1: {chunks[ranked_indices[0]][:50]}... (Score: {similarities[ranked_indices[0]]:.3f})")

# Expected: Correct chunk at rank #1 with high score

Inference Efficiency

With MRR optimization, you typically only need top 1-2 chunks :

# Efficient inference - high probability correct chunk is #1
top_chunk = chunks[similarities.argmax()]
confidence = similarities.max()

if confidence > 0.7:  # High confidence threshold
    return top_chunk  # Likely the correct answer
else:
    return chunks[similarities.argsort()[::-1][:3]]  # Return top 3 as fallback

Evaluation Methodology

• Training : train_question + val_question with MRR optimization
• Validation : MRR for early stopping, Recall@3/5 monitoring
• Test : test_question used once for final comparison
• Comparison : Direct evaluation against base E5-multilingual model
• Metrics : MRR, Recall@1,2,3,4,5, Hierarchy Rate

Perfect For

🎯 Educational Q&A : Exact answers at rank #1 consistently ⚡ Efficient Systems : Fewer chunks needed at inference
🇻🇳 Vietnamese Math : Specialized mathematical terminology 📊 Quality Ranking : Hierarchical relevance scoring 🚀 Production Ready : Proven improvement over base model

Technical Notes

• Base Model : intfloat/multilingual-e5-base
• Fine-tuning : Hierarchical contrastive learning with MRR optimization
• Max Sequence : 256 tokens
• Training Data : Vietnamese mathematical content with expert annotations
• Validation : Proper train/validation/test split methodology

Fine-tuned on 25/06/2025 with comprehensive base model comparison.