faisalmumtaz / codecompass-embed

CodeCompass-Embed

CodeCompass-Embed is a code embedding model fine-tuned from Qwen2.5-Coder-0.5B for semantic code search and retrieval tasks.

Model Highlights

• 🏆 #1 on CodeTrans-DL (code translation between frameworks)
• 🥇 #4 on CodeSearchNet-Python (natural language to code search)
• ⚡ 494M parameters, 896-dim embeddings
• 🔄 Bidirectional attention (converted from causal LLM)
• 🎯 Mean pooling with L2 normalization
• 📏 Trained at 512 tokens, extrapolates to longer sequences via RoPE

Model Details

Benchmark Results (CoIR)

Evaluated on the CoIR Benchmark (NDCG@10). Sorted by CSN-Python.

CodeCompass-Embed ranks #1 on CodeTrans-DL and #4 on CSN-Python.

Usage

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("faisalmumtaz/codecompass-embed", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("faisalmumtaz/codecompass-embed")

# Enable bidirectional attention
for layer in model.layers:
    layer.self_attn.is_causal = False

model.eval()

def encode(texts, is_query=False):
    if is_query:
        texts = [f"Instruct: Find the most relevant code snippet given the following query:
Query: {t}" for t in texts]
    
    inputs = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
    
    with torch.no_grad():
        outputs = model(**inputs, output_hidden_states=True)
        hidden = outputs.hidden_states[-1]
        mask = inputs["attention_mask"].unsqueeze(-1).float()
        embeddings = (hidden * mask).sum(1) / mask.sum(1).clamp(min=1e-9)
        embeddings = F.normalize(embeddings, p=2, dim=-1)
    
    return embeddings

query_emb = encode(["sort a list"], is_query=True)
code_embs = encode(["def sort(lst): return sorted(lst)"])
similarity = (query_emb @ code_embs.T).item()

Instruction Templates

Documents do not need instruction prefixes.

Training

• Data : 8.8M samples from CoRNStack, StackOverflow, CodeSearchNet
• Loss : InfoNCE (τ=0.05) with 7 hard negatives per sample
• Batch Size : 1024 (via GradCache)
• Steps : 950
• Hardware : NVIDIA H100

Limitations

• Weaker on Q&A style tasks (StackOverflow-QA, CodeFeedback)
• Trained on Python/JavaScript/Java/Go/PHP/Ruby

Citation

@misc{codecompass2026,
  author = {Faisal Mumtaz},
  title = {CodeCompass-Embed: A Code Embedding Model for Semantic Code Search},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/faisalmumtaz/codecompass-embed}
}

License

Apache 2.0