CATIE-AQ / CSR_Sparse_Encoder_camembert-large_STS

CSR Sparse Encoder

This is a CSR Sparse Encoder model finetuned from almanach/camembert-large on the french_sts dataset using the sentence-transformers library. It maps sentences & paragraphs to a 4096-dimensional sparse vector space with 256 maximum active dimensions and can be used for semantic search and sparse retrieval.

Model Details

Model Description

• Model Type: CSR Sparse Encoder
• Base model: almanach/camembert-large
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 4096 dimensions (trained with 256 maximum active dimensions)
• Similarity Function: Cosine Similarity
• Training Dataset:
  • french_sts
• Language: fr

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Sparse Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sparse Encoders on Hugging Face

Full Model Architecture

SparseEncoder(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'CamembertModel'})
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): SparseAutoEncoder({'input_dim': 1024, 'hidden_dim': 4096, 'k': 256, 'k_aux': 512, 'normalize': False, 'dead_threshold': 30})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("bourdoiscatie/SPLADE_camembert-large_STS")
# Run inference
sentences = [
    "Oui, je peux vous dire d'après mon expérience personnelle qu'ils ont certainement sifflé.",
    "Il est vrai que les bombes de la Seconde Guerre mondiale faisaient un bruit de sifflet lorsqu'elles tombaient.",
    "J'envisage de dépenser les 48 dollars par mois pour le système GTD (Getting things done) annoncé par David Allen.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 4096]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.3673, 0.2794],
#         [0.3673, 1.0000, 0.2023],
#         [0.2794, 0.2023, 1.0000]])

Evaluation

Metrics

Semantic Similarity

• Datasets: sts-dev and sts-test
• Evaluated with SparseEmbeddingSimilarityEvaluator

Training Details

Training Dataset

french_sts

• Dataset: french_sts at 47128cc
• Size: 12,227 training samples
• Columns: sentence1 , sentence2 , and score
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2	score
  type	string	string	float
  details	min: 6 tokens mean: 11.75 tokens max: 30 tokens	min: 6 tokens mean: 11.79 tokens max: 35 tokens	min: 0.0 mean: 0.44 max: 1.0

• Samples:

  sentence1	sentence2	score
  Un avion est en train de décoller.	Un avion est en train de décoller.	1.0
  Un homme est en train de fumer.	Un homme fait du patinage.	0.10000000149011612
  Une personne jette un chat au plafond.	Une personne jette un chat au plafond.	1.0

• Loss: SpladeLoss with these parameters:

  {
      "loss": "SparseCosineSimilarityLoss(loss_fct='torch.nn.modules.loss.MSELoss')",
      "document_regularizer_weight": 0.003
  }
  

Evaluation Dataset

french_sts

• Dataset: french_sts at 47128cc
• Size: 3,526 evaluation samples
• Columns: sentence1 , sentence2 , and score
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2	score
  type	string	string	float
  details	min: 6 tokens mean: 19.13 tokens max: 50 tokens	min: 6 tokens mean: 19.05 tokens max: 55 tokens	min: 0.0 mean: 0.43 max: 1.0

• Samples:

  sentence1	sentence2	score
  Un homme avec un casque de sécurité est en train de danser.	Un homme portant un casque de sécurité est en train de danser.	1.0
  Un jeune enfant monte à cheval.	Un enfant monte à cheval.	0.949999988079071
  Un homme donne une souris à un serpent.	L'homme donne une souris au serpent.	1.0

• Loss: SpladeLoss with these parameters:

  {
      "loss": "SparseCosineSimilarityLoss(loss_fct='torch.nn.modules.loss.MSELoss')",
      "document_regularizer_weight": 0.003
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy : epoch
• per_device_train_batch_size : 16
• per_device_eval_batch_size : 16
• bf16 : True

All Hyperparameters

Click to expand

• overwrite_output_dir : False
• do_predict : False
• eval_strategy : epoch
• prediction_loss_only : True
• per_device_train_batch_size : 16
• per_device_eval_batch_size : 16
• per_gpu_train_batch_size : None
• per_gpu_eval_batch_size : None
• gradient_accumulation_steps : 1
• eval_accumulation_steps : None
• torch_empty_cache_steps : None
• learning_rate : 5e-05
• weight_decay : 0.0
• adam_beta1 : 0.9
• adam_beta2 : 0.999
• adam_epsilon : 1e-08
• max_grad_norm : 1.0
• num_train_epochs : 3
• max_steps : -1
• lr_scheduler_type : linear
• lr_scheduler_kwargs : {}
• warmup_ratio : 0.0
• warmup_steps : 0
• log_level : passive
• log_level_replica : warning
• log_on_each_node : True
• logging_nan_inf_filter : True
• save_safetensors : True
• save_on_each_node : False
• save_only_model : False
• restore_callback_states_from_checkpoint : False
• no_cuda : False
• use_cpu : False
• use_mps_device : False
• seed : 42
• data_seed : None
• jit_mode_eval : False
• use_ipex : False
• bf16 : True
• fp16 : False
• fp16_opt_level : O1
• half_precision_backend : auto
• bf16_full_eval : False
• fp16_full_eval : False
• tf32 : None
• local_rank : 0
• ddp_backend : None
• tpu_num_cores : None
• tpu_metrics_debug : False
• debug : []
• dataloader_drop_last : False
• dataloader_num_workers : 0
• dataloader_prefetch_factor : None
• past_index : -1
• disable_tqdm : False
• remove_unused_columns : True
• label_names : None
• load_best_model_at_end : False
• ignore_data_skip : False
• fsdp : []
• fsdp_min_num_params : 0
• fsdp_config : {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size : 0
• fsdp_transformer_layer_cls_to_wrap : None
• accelerator_config : {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed : None
• label_smoothing_factor : 0.0
• optim : adamw_torch
• optim_args : None
• adafactor : False
• group_by_length : False
• length_column_name : length
• ddp_find_unused_parameters : None
• ddp_bucket_cap_mb : None
• ddp_broadcast_buffers : False
• dataloader_pin_memory : True
• dataloader_persistent_workers : False
• skip_memory_metrics : True
• use_legacy_prediction_loop : False
• push_to_hub : False
• resume_from_checkpoint : None
• hub_model_id : None
• hub_strategy : every_save
• hub_private_repo : None
• hub_always_push : False
• gradient_checkpointing : False
• gradient_checkpointing_kwargs : None
• include_inputs_for_metrics : False
• include_for_metrics : []
• eval_do_concat_batches : True
• fp16_backend : auto
• push_to_hub_model_id : None
• push_to_hub_organization : None
• mp_parameters :
• auto_find_batch_size : False
• full_determinism : False
• torchdynamo : None
• ray_scope : last
• ddp_timeout : 1800
• torch_compile : False
• torch_compile_backend : None
• torch_compile_mode : None
• include_tokens_per_second : False
• include_num_input_tokens_seen : False
• neftune_noise_alpha : None
• optim_target_modules : None
• batch_eval_metrics : False
• eval_on_start : False
• use_liger_kernel : False
• eval_use_gather_object : False
• average_tokens_across_devices : False
• prompts : None
• batch_sampler : batch_sampler
• multi_dataset_batch_sampler : proportional
• router_mapping : {}
• learning_rate_mapping : {}

Training Logs

Framework Versions

• Python: 3.12.3
• Sentence Transformers: 5.0.0
• Transformers: 4.51.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.6.0
• Datasets: 2.16.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

SpladeLoss

@misc{formal2022distillationhardnegativesampling,
      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
      year={2022},
      eprint={2205.04733},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2205.04733},
}

FlopsLoss

@article{paria2020minimizing,
    title={Minimizing flops to learn efficient sparse representations},
    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
    journal={arXiv preprint arXiv:2004.05665},
    year={2020}
}