yasserrmd / kallamni-embed-v1

kallamni-embed-v1 — Emirati Spoken Arabic Embedding Model

Author: @yasserrmd
Version: v1
License: Apache 2.0

---

🎯 Motivation

kallamni-embed-v1 was built to address a gap in Arabic NLP — the absence of a high-fidelity model for spoken Emirati Arabic .
While most Arabic embeddings (AraBERT, CAMeLBERT, MARBERT) focus on MSA or pan-Arab dialects , they fail to capture UAE’s informal patterns such as:

• Lexical variants: وايد , مب , سير , ويّاكم
• Code-switching: “bro yalla lets go al mall”
• Arabizi + emojis: “ana mb 3arf 😅 sho y9eer!”

This model learns these naturally occurring forms using curated Emirati-style Q&A and conversation datasets.

SentenceTransformer based on BAAI/bge-m3

This is a sentence-transformers model finetuned from BAAI/bge-m3 . It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-m3
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'كيف كانت التجربة في المطعم اليديد؟',
    'المطعم كان ممتاز، الأكل لذيذ والخدمة سريعة.',
    'كنت وايد سعيد، السوالف ما خلصت بيننا.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

• Size: 50,000 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 7 tokens mean: 13.47 tokens max: 24 tokens	min: 8 tokens mean: 18.85 tokens max: 36 tokens

• Samples:

  sentence_0	sentence_1
  قد استخدمت تطبيق تتبع السعرات الحرارية؟	إيه، يساعدني في مراقبة أكلي ونسبة البروتين.
  شو كانت أول تجربة لك في التدريب العملي؟	كانت مميزة، استفدت وتعلمت أشياء ما تدرسها الكتب.
  إذا حد قال 'على عينه حار'، شو يقصد؟	يعني هذا شخص صريح وما يجامل، يقول اللي في قلبه.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size : 24
• per_device_eval_batch_size : 24
• fp16 : True
• multi_dataset_batch_sampler : round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir : False
• do_predict : False
• eval_strategy : no
• prediction_loss_only : True
• per_device_train_batch_size : 24
• per_device_eval_batch_size : 24
• per_gpu_train_batch_size : None
• per_gpu_eval_batch_size : None
• gradient_accumulation_steps : 1
• eval_accumulation_steps : None
• torch_empty_cache_steps : None
• learning_rate : 5e-05
• weight_decay : 0.0
• adam_beta1 : 0.9
• adam_beta2 : 0.999
• adam_epsilon : 1e-08
• max_grad_norm : 1
• num_train_epochs : 3
• max_steps : -1
• lr_scheduler_type : linear
• lr_scheduler_kwargs : {}
• warmup_ratio : 0.0
• warmup_steps : 0
• log_level : passive
• log_level_replica : warning
• log_on_each_node : True
• logging_nan_inf_filter : True
• save_safetensors : True
• save_on_each_node : False
• save_only_model : False
• restore_callback_states_from_checkpoint : False
• no_cuda : False
• use_cpu : False
• use_mps_device : False
• seed : 42
• data_seed : None
• jit_mode_eval : False
• use_ipex : False
• bf16 : False
• fp16 : True
• fp16_opt_level : O1
• half_precision_backend : auto
• bf16_full_eval : False
• fp16_full_eval : False
• tf32 : None
• local_rank : 1
• ddp_backend : None
• tpu_num_cores : None
• tpu_metrics_debug : False
• debug : []
• dataloader_drop_last : True
• dataloader_num_workers : 0
• dataloader_prefetch_factor : None
• past_index : -1
• disable_tqdm : False
• remove_unused_columns : True
• label_names : None
• load_best_model_at_end : False
• ignore_data_skip : False
• fsdp : []
• fsdp_min_num_params : 0
• fsdp_config : {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap : None
• accelerator_config : {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed : None
• label_smoothing_factor : 0.0
• optim : adamw_torch
• optim_args : None
• adafactor : False
• group_by_length : False
• length_column_name : length
• ddp_find_unused_parameters : None
• ddp_bucket_cap_mb : None
• ddp_broadcast_buffers : False
• dataloader_pin_memory : True
• dataloader_persistent_workers : False
• skip_memory_metrics : True
• use_legacy_prediction_loop : False
• push_to_hub : False
• resume_from_checkpoint : None
• hub_model_id : None
• hub_strategy : every_save
• hub_private_repo : None
• hub_always_push : False
• gradient_checkpointing : False
• gradient_checkpointing_kwargs : None
• include_inputs_for_metrics : False
• include_for_metrics : []
• eval_do_concat_batches : True
• fp16_backend : auto
• push_to_hub_model_id : None
• push_to_hub_organization : None
• mp_parameters :
• auto_find_batch_size : False
• full_determinism : False
• torchdynamo : None
• ray_scope : last
• ddp_timeout : 1800
• torch_compile : False
• torch_compile_backend : None
• torch_compile_mode : None
• include_tokens_per_second : False
• include_num_input_tokens_seen : False
• neftune_noise_alpha : None
• optim_target_modules : None
• batch_eval_metrics : False
• eval_on_start : False
• use_liger_kernel : False
• eval_use_gather_object : False
• average_tokens_across_devices : False
• prompts : None
• batch_sampler : batch_sampler
• multi_dataset_batch_sampler : round_robin

Training Logs

---

Evaluation Overview

V4 — Hyper-Authentic Emirati Benchmark

→ +57 % gain in retrieval relevance over the multilingual baseline.

---

V5 — Dialect Robustness Benchmark

Statistical significance: Δ nDCG@10 = +0.0218 (95 % CI [0.0008 – 0.0439], p = 0.04)

---

📈 Visual Summary

The Emirati-tuned model maintains high stability across dialectal noise — especially Arabizi , Code-Switch , and Social Noise subsets — where multilingual models collapse.

---

🧠 Robustness & Use Cases

• Handles informal input: Arabizi, emojis, typos, and Gulf-accented syntax.
• Optimized for retrieval & RAG: Works well in vector databases for Emirati chatbots, citizen-service platforms, and multilingual UAE apps.
• Fast inference: ~15 % faster than multilingual-e5-large on average batch size 32.
• Cross-dialect adaptability: Maintains coherence on Gulf-neighbor variations (Kuwaiti, Omani).

---

🧩 Why Other Models Were Excluded

These models were retained for reference but excluded from the final leaderboard because they lack UAE-specific conversational grounding .

---

🔬 Benchmark Protocol

All datasets were auto-synthesized inside the evaluation script to ensure control and reproducibility.

• Retrieval pairs: 500 queries × 500 docs (3 hard negatives per gold)
• Similarity pairs: 2 000 sentence pairs
• Classification: 3 600 texts across 3 classes (Complaint / Humor / Question)
• 5-fold cross-validation + paired bootstrap CIs

---

Intended Use

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 4.1.0
• Transformers: 4.52.4
• PyTorch: 2.6.0+cu124
• Accelerate: 1.8.1
• Datasets: 3.6.0
• Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}