Shuu12121 / CodeSearch-ModernBERT-Owl-5.2

SentenceTransformer based on Shuu12121/CodeModernBERT-Owl-5.2

This is a sentence-transformers model finetuned from Shuu12121/CodeModernBERT-Owl-5.2 . It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Shuu12121/CodeModernBERT-Owl-5.2
• Maximum Sequence Length: 1024 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    "WalkShallow reads the entries in the named directory and calls fn for each.\nIt does not recurse into subdirectories.\n\nIf fn returns an error, iteration stops and WalkShallow returns that value.\n\nOn Linux, WalkShallow does not allocate, so long as certain methods on the\nWalkFunc's DirEntry are not called which necessarily allocate.",
    'func WalkShallow(dirName mem.RO, fn WalkFunc) error {\n\tif f := osWalkShallow; f != nil {\n\t\treturn f(dirName, fn)\n\t}\n\tof, err := os.Open(dirName.StringCopy())\n\tif err != nil {\n\t\treturn err\n\t}\n\tdefer of.Close()\n\tfor {\n\t\tfis, err := of.ReadDir(100)\n\t\tfor _, de := range fis {\n\t\t\tif err := fn(mem.S(de.Name()), de); err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t}\n\t\tif err != nil {\n\t\t\tif err == io.EOF {\n\t\t\t\treturn nil\n\t\t\t}\n\t\t\treturn err\n\t\t}\n\t}\n}',
    '@Before\n  public void setupMockCluster() throws IOException {\n    Configuration conf = new HdfsConfiguration();\n    conf.setDouble(DFSConfigKeys.DFS_NAMENODE_SAFEMODE_THRESHOLD_PCT_KEY,\n        THRESHOLD);\n    conf.setInt(DFSConfigKeys.DFS_NAMENODE_SAFEMODE_EXTENSION_KEY,\n        EXTENSION);\n    conf.setInt(DFSConfigKeys.DFS_NAMENODE_SAFEMODE_MIN_DATANODES_KEY,\n        DATANODE_NUM);\n\n    fsn = mock(FSNamesystem.class);\n    doReturn(true).when(fsn).hasWriteLock();\n    doReturn(true).when(fsn).hasReadLock();\n    doReturn(true).when(fsn).hasWriteLock(RwLockMode.BM);\n    doReturn(true).when(fsn).hasReadLock(RwLockMode.BM);\n    doReturn(true).when(fsn).isRunning();\n    NameNode.initMetrics(conf, NamenodeRole.NAMENODE);\n\n    bm = spy(new BlockManager(fsn, false, conf));\n    doReturn(true).when(bm).isGenStampInFuture(any(Block.class));\n    dn = spy(bm.getDatanodeManager());\n    Whitebox.setInternalState(bm, "datanodeManager", dn);\n    // the datanode threshold is always met\n    when(dn.getNumLiveDataNodes()).thenReturn(DATANODE_NUM);\n\n    bmSafeMode = new BlockManagerSafeMode(bm, fsn, false, conf);\n  }',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9081, 0.0440],
#         [0.9081, 1.0000, 0.0554],
#         [0.0440, 0.0554, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 4,000,000 training samples
• Columns: sentence_0 , sentence_1 , and label
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	float
  details	min: 8 tokens mean: 76.07 tokens max: 1024 tokens	min: 13 tokens mean: 150.67 tokens max: 1024 tokens	min: 1.0 mean: 1.0 max: 1.0

• Samples:

  sentence_0	sentence_1	label
  Set the column title @param column - column number (first column is: 0) @param title - new column title	setHeader = function(column, newValue) { const obj = this; if (obj.headers[column]) { const oldValue = obj.headers[column].textContent; const onchangeheaderOldValue = (obj.options.columns && obj.options.columns[column] && obj.options.columns[column].title)
  Elsewhere this is known as a "Weak Value Map". Whereas a std JS WeakMap is weak on its keys, this map is weak on its values. It does not retain these values strongly. If a given value disappears, then the entries for it disappear from every weak-value-map that holds it as a value. Just as a WeakMap only allows gc-able values as keys, a weak-value-map only allows gc-able values as values. Unlike a WeakMap, a weak-value-map unavoidably exposes the non-determinism of gc to its clients. Thus, both the ability to create one, as well as each created one, must be treated as dangerous capabilities that must be closely held. A program with access to these can read side channels though gc that do not* rely on the ability to measure duration. This is a separate, and bad, timing-independent side channel. This non-determinism also enables code to escape deterministic replay. In a blockchain context, this could cause validators to differ from each other, preventing consensus, and thus preventing ...	makeFinalizingMap = (finalizer, opts) => { const { weakValues = false } = opts
  Creates a function that memoizes the result of func . If resolver is provided, it determines the cache key for storing the result based on the arguments provided to the memoized function. By default, the first argument provided to the memoized function is used as the map cache key. The func is invoked with the this binding of the memoized function. Note: The cache is exposed as the cache property on the memoized function. Its creation may be customized by replacing the _.memoize.Cache constructor with one whose instances implement the Map method interface of delete , get , has , and set . @static @memberOf _ @since 0.1.0 @category Function @param {Function} func The function to have its output memoized. @param {Function} [resolver] The function to resolve the cache key. @returns {Function} Returns the new memoized function. @example var object = { 'a': 1, 'b': 2 }; var othe...	function memoize(func, resolver) { if (typeof func != 'function'

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size : 200
• per_device_eval_batch_size : 200
• fp16 : True
• multi_dataset_batch_sampler : round_robin

All Hyperparameters

Training Logs

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 5.0.0
• Transformers: 4.53.1
• PyTorch: 2.7.0+cu128
• Accelerate: 1.7.0
• Datasets: 3.6.0
• Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}