tomaarsen / ModernBERT-base-msmarco-margin-mse

ModernBERT-base finetuned on MSMARCO via distillation

This is a sentence-transformers model finetuned from answerdotai/ModernBERT-base on the msmarco-qwen3-reranker-0.6_b dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: answerdotai/ModernBERT-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • msmarco-qwen3-reranker-0.6_b
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("tomaarsen/ModernBERT-base-msmarco-margin-mse")
# Run inference
queries = [
    "what does ly mean in a blood test",
]
documents = [
    'According to the Hormone-Refractory Prostate Cancer Association, LY on a blood test stands for lymphocytes. The number in the results represents the percentage of lymphocytes in the white blood count. Lymphocytes should count for 15 to 46.8 percent of white blood cells. Continue Reading.',
    "FROM OUR COMMUNITY. Hi Terry, The LY (Lymphocytes) in your blood test is; the type of white blood cell found in the blood and lymph systems; part of the immune system. BUN/CREAT - Bun and Creatinine are tests done to monitor kidney function. I'm sorry, but I've never heard of the other 2.",
    'FROM OUR EXPERTS. Trace lysed blood refers to a finding that is usually reported from a urinary dip stick analysis. It implies that there is a small quantity of red cells in the urine that have broken open. The developer on the dip stick reacts with the hemoglobin that is released when the red cells are lysed.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.9409, 0.9409, 0.9366]])

Evaluation

Metrics

Information Retrieval

• Datasets: msmarco-eval-1kq-1kd , NanoMSMARCO , NanoNFCorpus and NanoNQ
• Evaluated with InformationRetrievalEvaluator

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with NanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ]
  }
  

Training Details

Training Dataset

msmarco-qwen3-reranker-0.6_b

• Dataset: msmarco-qwen3-reranker-0.6_b at 20c25c8
• Size: 10,000 training samples
• Columns: query , positive , negative_1 , negative_2 , negative_3 , negative_4 , negative_5 , negative_6 , negative_7 , negative_8 , and score
• Approximate statistics based on the first 1000 samples:

  	query	positive	negative_1	negative_2	negative_3	negative_4	negative_5	negative_6	negative_7	negative_8	score
  type	string	string	string	string	string	string	string	string	string	string	list
  details	min: 5 tokens mean: 9.32 tokens max: 40 tokens	min: 17 tokens mean: 81.37 tokens max: 288 tokens	min: 16 tokens mean: 68.92 tokens max: 208 tokens	min: 12 tokens mean: 70.8 tokens max: 204 tokens	min: 15 tokens mean: 70.5 tokens max: 260 tokens	min: 16 tokens mean: 71.64 tokens max: 244 tokens	min: 13 tokens mean: 72.57 tokens max: 226 tokens	min: 11 tokens mean: 72.34 tokens max: 198 tokens	min: 15 tokens mean: 73.57 tokens max: 202 tokens	min: 15 tokens mean: 71.21 tokens max: 200 tokens	size: 9 elements

• Samples:

  query	positive	negative_1	negative_2	negative_3	negative_4	negative_5	negative_6	negative_7	negative_8	score
  what is clomiphene	Uses of This Medicine. Clomiphene is used as a fertility medicine in some women who are unable to become pregnant. Clomiphene probably works by changing the hormone balance of the body. In women, this causes ovulation to occur and prepares the body for pregnancy.ses of This Medicine. Clomiphene is used as a fertility medicine in some women who are unable to become pregnant. Clomiphene probably works by changing the hormone balance of the body. In women, this causes ovulation to occur and prepares the body for pregnancy.	Clomiphene citrate, a synthetic hormone commonly used to induce or regulate ovulation, is the most often prescribed fertility pill. Brand names for clomiphene citrate include Clomid and Serophene. Clomiphene works indirectly to stimulate ovulation.	Occasionally, clomiphene can stimulate the ovaries too much, causing multiple eggs to be released, which can result in multiple births, such as twins or triplets (see Clomid and Twins) . Clomiphene is one of the least expensive and easiest-to-use fertility drugs. However, it will not work for all types of infertility. Your healthcare provider needs to try to find your cause of infertility before you try clomiphene.	Clomiphene Citrate offers two benefits to the performance enhancing athlete with one being primary. Most commonly, this SERM is used for post cycle recovery purposes; specifically to stimulate natural testosterone production that has been suppressed due to the use of anabolic steroids.	PCOS and ovulation problems and Clomid treatment. Clomid (clomiphene citrate or Serophene) is an oral medication that is commonly used for the treatment of infertility. It is often given to try to induce ovulation in women that do not develop and release an egg (ovulate) on their own.	Indication: Clomid (clomiphene citrate) is often the first choice for treating infertility, because it's effective and been used for more than 40 years.	Clomid Description. 1 Clomid (clomiphene citrate tablets USP) is an orally administered, nonsteroidal, ovulatory stimulant designated chemically as 2-[p-(2-chloro-1,2-diphenylvinyl)phenoxy] triethylamine citrate (1:1). It has the molecular formula of C26H28ClNO • C6H8O7 and a molecular weight of 598.09.	PCOS and ovulation problems and Clomid treatment. Clomid (clomiphene citrate or Serophene) is an oral medication that is commonly used for the treatment of infertility. 1 It is often given to try to induce ovulation in women that do not develop and release an egg (ovulate) on their own. Clomid is started early in the menstrual cycle and is taken for five days either from cycle days 3 through 7, or from day 5 through 9. 2 Clomid is usually started at a dose of one tablet (50mg) daily-taken any time of day.	Clomid is taken as a pill. This is unlike the stronger fertility drugs, which require injection. Clomid is also very effective, stimulating ovulation 80 percent of the time. Clomid may also be marketed under the name Serophene, or you may see it sold under its generic name, clomiphene citrate. Note: Clomid can also be used as a treatment for male infertility. This article focuses on Clomid treatment in women.	[4.75390625, 6.9375, 3.92578125, 1.0400390625, 5.61328125, ...]
  typical accountant cost for it contractor	In the current market, we’ve seen rates as low as £50 +VAT, and as high as £180 +VAT for dedicated contractor accountants. Interestingly, the average cost of contractor accounting has not risen in line with inflation over the past decade.	So, how much does a contractor cost, anywhere from 5% to 25% of the total project cost, with the average ranging 10-15%.ypically the contractor' s crew will be general carpentry trades people, some who may have more specialized skills. Exactly how a general contractor charges for a project depends on the type of contract you agree to. There are three common types of cost contracts, fixed price, time & materials and cost plus a fee.	1 Accountants charge $150-$400 or more an hour, depending on the type of work, the size of the firm and its location. 2 You'll pay lower rates for routine work done by a less-experienced associate or lesser-trained employee, such as $30-$50 for bookkeeping services. 3 An accountant's total fee depends on the project. For a simple start-up, expect a minimum of 0.5-1.5 hours of consultation ($75-$600) to go over your business structure and basic tax issues.	So, how much does a contractor cost, anywhere from 5% to 25% of the total project cost, with the average ranging 10-15%.xactly how a general contractor charges for a project depends on the type of contract you agree to. There are three common types of cost contracts, fixed price, time & materials and cost plus a fee. Each contract type has pros and cons for both the consumer and for the contractor.	1 Accountants charge $150-$400 or more an hour, depending on the type of work, the size of the firm and its location. 2 You'll pay lower rates for routine work done by a less-experienced associate or lesser-trained employee, such as $30-$50 for bookkeeping services. 3 An accountant's total fee depends on the project.	average data entry keystrokes per hour salaries the average salary for data entry keystrokes per hour jobs is $ 20000	Accounting services are typically $250 to $400 per month, or $350 to $500 per quarter. Sales tax and bank recs included. We do all the processing, filing and tax deposits. 5 employees, bi-weekly payroll, direct deposit, $135 per month.	The less that is outsourced, the cheaper it will be for you. A bookkeeper should be paid between $15 and $18 per hour. An accountant with a undergraduate degree (4-years) should be paid somewhere around $20/hour but that still depends on what you're having them do. An accountant with a graduate degree (masters) should be paid between $25 and $30 per hour.	Pay by Experience Level for Intelligence Analyst. Median of all compensation (including tips, bonus, and overtime) by years of experience. Intelligence Analysts with a lot of experience tend to enjoy higher earnings.	[7.44921875, 3.271484375, 5.859375, 3.234375, 5.421875, ...]
  what is mch on a blood test	What High Levels Mean. MCH levels in blood tests are considered high if they are 35 or higher. A normal hemoglobin level is considered to be in the range between 26 and 33 picograms per red blood cell. High MCH levels can indicate macrocytic anemia, which can be caused by insufficient vitamin B12.acrocytic RBCs are large so tend to have a higher MCH, while microcytic red cells would have a lower value.”. MCH is one of three red blood cell indices (MCHC and MCV are the other two). The measurements are done by machine and can help with diagnosis of medical problems.	MCH stands for mean corpuscular hemoglobin. It estimates the average amount of hemoglobin in each red blood cell, measured in picograms (a trillionth of a gram). Automated cell counters calculate the MCH, which is reported as part of a complete blood count (CBC) test. MCH may be low in iron-deficiency anemia, and may be high in anemia due to vitamin B12 or folate deficiency. Other forms of anemia can also cause MCH to be abnormal. Doctors only use the MCH as supporting information, not to make a diagnosis.	A. MCH stands for mean corpuscular hemoglobin. It estimates the average amount of hemoglobin in each red blood cell, measured in picograms (a trillionth of a gram). Automated cell counters calculate the MCH, which is reported as part of a complete blood count (CBC) test. MCH may be low in iron-deficiency anemia, and may be high in anemia due to vitamin B12 or folate deficiency. Other forms of anemia can also cause MCH to be abnormal.	The test used to determine the quantity of hemoglobin in the blood is known as the MCH blood test. The full form of MCH is Mean Corpuscular Hemoglobin. This test is therefore used to determine the average amount of hemoglobin per red blood cell in the body. The results of the MCH blood test are therefore reported in picograms, a tiny measure of weight.	MCH blood test high indicates that there is a poor supply of oxygen to the blood where as MCH blood test low mean that hemoglobin is too little in the cells indicating a lack of iron. It is important that iron is maintained at a certain level as too much or too little iron can be dangerous to your body.	slide 1 of 7. What Is MCH? MCH is the initialism for Mean Corpuscular Hemoglobin. Taken from Latin, the term refers to the average amount of hemoglobin found in red blood cells. A CBC (complete blood count) blood test can be used to monitor MCH levels in the blood. Lab Tests Online explains that the MCH aspect of a CBC test “is a measurement of the average amount of oxygen-carrying hemoglobin inside a red blood cell. Macrocytic RBCs are large so tend to have a higher MCH, while microcytic red cells would have a lower value..	The test used to determine the quantity of hemoglobin in the blood is known as the MCH blood test. The full form of MCH is Mean Corpuscular Hemoglobin. This test is therefore used to determine the average amount of hemoglobin per red blood cell in the body. The results of the MCH blood test are therefore reported in picograms, a tiny measure of weight. The normal range of the MCH blood test is between 26 and 33 pg per cell.	A MCHC test is a test that is carried out to test a person for anemia. The MCHC in a MCHC test stands for Mean Corpuscular Hemoglobin Concentration. MCHC is the calculation of the average hemoglobin inside a red blood cell. A MCHC test can be performed along with a MCV test (Mean Corpuscular Volume).Both levels are used to test people for anemia.The MCHC test is also known as the MCH blood test which tests the levels of hemoglobin in the blood. The MCHC test can be ordered as part of a complete blood count (CBC) test.CHC is measured in grams per deciliter. Normal readings for MCHC are 31 grams per deciliter to 35 grams per deciliter. A MCHC blood test may be ordered when a person is showing signs of fatigue or weakness, when there is an infection, is bleeding or bruising easily or when there is an inflammation.	The test looks at the average amount of hemoglobin per red cell. So MCHC = the amount of hemoglobin present in each red blood cell. A MCHC blood test could be ordered for someone who has signs of fatigue or weakness, when there is an infection, is bleeding or bruising easily or when there is noticeable inflammation.	[6.44921875, 7.05078125, 7.2109375, 8.40625, 6.53515625, ...]

• Loss: MarginMSELoss

Evaluation Dataset

msmarco-qwen3-reranker-0.6_b

• Dataset: msmarco-qwen3-reranker-0.6_b at 20c25c8
• Size: 1,000 evaluation samples
• Columns: query , positive , negative_1 , negative_2 , negative_3 , negative_4 , negative_5 , negative_6 , negative_7 , negative_8 , and score
• Approximate statistics based on the first 1000 samples:

  	query	positive	negative_1	negative_2	negative_3	negative_4	negative_5	negative_6	negative_7	negative_8	score
  type	string	string	string	string	string	string	string	string	string	string	list
  details	min: 4 tokens mean: 9.27 tokens max: 24 tokens	min: 22 tokens mean: 82.29 tokens max: 236 tokens	min: 11 tokens mean: 69.82 tokens max: 215 tokens	min: 11 tokens mean: 69.4 tokens max: 209 tokens	min: 16 tokens mean: 71.47 tokens max: 223 tokens	min: 14 tokens mean: 70.96 tokens max: 225 tokens	min: 14 tokens mean: 73.09 tokens max: 218 tokens	min: 16 tokens mean: 69.31 tokens max: 215 tokens	min: 14 tokens mean: 72.3 tokens max: 226 tokens	min: 11 tokens mean: 70.66 tokens max: 210 tokens	size: 9 elements

• Samples:

  query	positive	negative_1	negative_2	negative_3	negative_4	negative_5	negative_6	negative_7	negative_8	score
  how many people employed by shell	Shell worldwide. Royal Dutch Shell was formed in 1907, although our history dates back to the early 19th century, to a small shop in London where the Samuel family sold sea shells. Today, Shell is one of the world’s major energy companies, employing an average of 93,000 people and operating in more than 70 countries. Our headquarters are in The Hague, the Netherlands, and our Chief Executive Officer is Ben van Beurden.	Show sources information. This statistic shows the number of employees at SeaWorld Entertainment, Inc. in the United States, by type. As of December 2016, SeaWorld employed 5,000 full-time employees and counted approximately 13,000 seasonal employees during their peak operating season.	Jobs, companies, people, and articles for LinkedIn’s Payroll Specialist - Addus Homecare, Inc. members. Insights about Payroll Specialist - Addus Homecare, Inc. members on LinkedIn. Median salary $31,300.	As of July 2014, there are 139 million people employed in the United States. This number is up by 209,000 employees from June and by 1.47 million from the beginning of 2014.	average data entry keystrokes per hour salaries the average salary for data entry keystrokes per hour jobs is $ 20000	Research and review Plano Synergy jobs. Learn more about a career with Plano Synergy including all recent jobs, hiring trends, salaries, work environment and more. Find Jobs Company Reviews Find Salaries Find Resumes Employers / Post Job Upload your resume Sign in	From millions of real job salary data. 13 Customer Support Specialist salary data. Average Customer Support Specialist salary is $59,032 Detailed Customer Support Specialist starting salary, median salary, pay scale, bonus data report Register & Know how much $ you can earn | Sign In	From millions of real job salary data. 1 Ceo Ally salary data. Average Ceo Ally salary is $55,000 Detailed Ceo Ally starting salary, median salary, pay scale, bonus data report	HelpSystems benefits and perks, including insurance benefits, retirement benefits, and vacation policy. Reported anonymously by HelpSystems employees. Glassdoor uses cookies to improve your site experience.	[6.265625, -1.3671875, -6.91796875, 1.111328125, -7.96875, ...]
  what is a lcsw	LCSW is an acronym for licensed clinical social worker, and people with this title are skilled professionals who meet certain requirements and work in a variety of fields. The term social worker is not always synonymous with licensed clinical social worker.	LISW means the person is a Licensed Independent Social Worker. LCSW means the person is a Licensed Clinical Social Worker. Source(s): Introduction to Social Work 101 at University of Nevada, Las Vega (UNLV) Dorothy K. · 1 decade ago.	An LCSW is a licensed clinical social worker. A LMHC is the newest addition to the field of mental health. They are highly similar and can do most of the same things with few exceptions. One thing to keep in mind is that because the LMHC lincense is so new, there are fewer in number in the field.n LCSW is a licensed clinical social worker. A LMHC is the newest addition to the field of mental health. They are highly similar and can do most of the same things with few exceptions. One thing to keep in mind is that because the LMHC lincense is so new, there are fewer in number in the field.	The Licensed Clinical Social Worker or LCSW, is a sub-sector within the field of Social Work. They work with clients in order to help them deal with issues involving their mental and emotional health. This could be related to substance abuse, past trauma or mental illness.	Licensed Clinical Social Worker | LCSW. The Licensed Clinical Social Worker or LCSW, is a sub-sector within the field of Social Work. LCSW's work with clients in order to help deal with issues involving mental and emotional health. There are a wide variety of specializations the Licensed Clinical Social Worker can focus on.	The LMSW exam is a computer-based test containing 170 multiple-choice questions designed to measure minimum competencies in four categories of social work practice: Human development, diversity, and behavior in the environment. Assessment and intervention planning.	The Licensed Clinical Social Worker, also known as the LCSW, is a branch of social work that specializes in mental health therapy in a counseling format. Becoming an LCSW requires a significant degree of training, including having earned a Master of Social Work (MSW) degree from a Council on Social Work Education (CSWE) accredited program.	a. The examination requirements for licensure as an LCSW include passing the Clinical Examination of the ASWB or the Clinical Social Workers Examination of the State of California. Scope of practice-Limitations. a.To the extent they are prepared through education and training, an LCSW can engage in all acts and practices defined as the practice of clinical social work. Certified Social Work (CSW): CSW means a licensed certified social worker. A CSW must have a master s degree.	The LTCM Client is a way for companies to stay in touch with you, their customers, in a way that is unobtrusive and completely under the users' control. It's an application that runs quietly on the computer. Users can and should customize the client to match their desired preferences.	[7.34375, 6.046875, 7.09765625, 6.46484375, 7.28515625, ...]
  does oolong tea have much caffeine?	At a given weight, tea contains more caffeine than coffee, but this doesn’t mean that a usual portion of tea contains more caffeine than coffee because tea is usually brewed in a weak way. Some kinds of tea, such as oolong and black tea, contain higher level of caffeine than most other teas. Among six basic teas (green, black, yellow, white, oolong, dark), green tea contains less caffeine than black tea and white tea contains less than green tea. But many studies found that the caffeine content varies more among individual teas than it does among broad categories.	Actually, oolong tea has less caffeine than coffee and black tea. A cup of oolong tea only has about 1/3 of caffeine of a cup of coffee. According to a research conducted by HICKS M.B, the caffeine decreases whenever the tea leaves go through the process of brewing.	Oolong tea contains caffeine. Caffeine works by stimulating the central nervous system (CNS), heart, and muscles. Oolong tea also contains theophylline and theobromine, which are chemicals similar to caffeine. Too much oolong tea, more than five cups per day, can cause side effects because of the caffeine.	Oolong tea, made from more mature leaves, usually have less caffeine than green tea. On the flip side, mature leaves contain less theanine, a sweet, natural relaxant that makes a tea much less caffeinated than it actually is. That is the theory, anyway.	Oolong tea is a product made from the leaves, buds, and stems of the Camellia sinensis plant. This is the same plant that is also used to make black tea and green tea. The difference is in the processing.Oolong tea is partially fermented, black tea is fully fermented, and green tea is unfermented. Oolong tea is used to sharpen thinking skills and improve mental alertness. It is also used to prevent cancer, tooth decay, osteoporosis, and heart disease.owever, do not drink more than 2 cups a day of oolong tea. That amount of tea contains about 200 mg of caffeine. Too much caffeine during pregnancy might cause premature delivery, low birth weight, and harm to the baby.	A Department of Nutritional Services report provides the following ranges of caffeine content for a cup of tea made with loose leaves: 1 Black Tea: 23 - 110 mg. 2 Oolong Tea: 12 - 55 mg. Green Tea: 8 - 36 mg.	Oolong tea is a product made from the leaves, buds, and stems of the Camellia sinensis plant. This is the same plant that is also used to make black tea and green tea. The difference is in the processing. Oolong tea is partially fermented, black tea is fully fermented, and green tea is unfermented. Oolong tea is used to sharpen thinking skills and improve mental alertness. It is also used to prevent cancer, tooth decay, osteoporosis, and heart disease.	Health Effects of Tea – Caffeine. In dry form, a kilogram of black tea has twice the caffeine as a kilogram of coffee…. But one kilogram of black tea makes about 450 cups of tea and one kilogram of coffee makes about 100 cups of coffee, so…. There is less caffeine in a cup of tea than in a cup of coffee. Green teas have less caffeine than black teas, and white teas have even less caffeine than green teas. Oolong teas fall between black and green teas. Herbal tea, because it is not made from the same tea plant, is caffeine-free, naturally. Here is a graphical representation of their respective caffeine content.	The average 8-ounce serving of brewed black tea contains 14 to 70 mg of caffeine. This compares to 24 to 45 mg of caffeine found in green tea. An 8-ounce glass of instant iced tea prepared with water contains 11 to 47 mg of caffeine. Most ready-to-drink bottled teas contain 5 to 40 mg of caffeine. Just as with coffee, decaffeinated tea still contains 5 to 10 mg of caffeine per cup.	[7.60546875, 8.78125, 9.109375, 8.609375, 7.984375, ...]

• Loss: MarginMSELoss

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy : steps
• per_device_train_batch_size : 16
• per_device_eval_batch_size : 16
• learning_rate : 4e-05
• num_train_epochs : 1
• warmup_ratio : 0.1
• bf16 : True

All Hyperparameters

Training Logs

Framework Versions

• Python: 3.11.10
• Sentence Transformers: 5.1.0.dev0
• Transformers: 4.51.2
• PyTorch: 2.5.1+cu124
• Accelerate: 1.5.2
• Datasets: 3.5.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MarginMSELoss

@misc{hofstätter2021improving,
    title={Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation},
    author={Sebastian Hofstätter and Sophia Althammer and Michael Schröder and Mete Sertkan and Allan Hanbury},
    year={2021},
    eprint={2010.02666},
    archivePrefix={arXiv},
    primaryClass={cs.IR}
}