bench-labs / cagliostro-v3.5-chat

cagliostro-v3.5-chat

The chat version of cagliostro-v3.5 , a 146M parameter language model. It is v3.5 after one epoch of supervised fine-tuning on smol-smoltalk , the conversation set SmolLM2-135M-Instruct and SmolLM2-360M-Instruct were tuned on. The architecture and tokenizer are unchanged. Only the weights are different.

For how this model and its base compare with SmolLM2-135M outside the leaderboard, see How it compares with SmolLM2-135M .

What the fine-tune does

The base model already saw SmolTalk conversations during pretraining, written as plain User: and Assistant: lines. This fine-tune keeps that format and teaches the model to end its turn. Every assistant reply ends with <|endoftext|> , so generation stops on its own instead of running on into the user's next message. Loss is computed on the assistant replies only.

Same 12 prompts for both models, greedy decoding, at most 200 new tokens:

The four chat replies that did not stop were long answers cut off by the 200 token limit. Every output from both models is in samples.json .

Loss on held-out assistant tokens (2,000 conversations from the smol-smoltalk test split):

How it compares with SmolLM2-135M

The Open SLM Leaderboard reports five benchmarks, so we also ran a wider set against SmolLM2-135M, the model cagliostro-v3.5 replaced at the top of that board, and against SmolLM2-135M-Instruct. Both sides got the same harness and the same settings. Every difference comes with a 95% interval from a paired bootstrap over the benchmark's items. All raw outputs are in evals/ .

Base models

cagliostro-v3.5 against SmolLM2-135M: lm-evaluation-harness 0.4.13, zero-shot, float32, full test sets, 2,048-token context for both. The first four rows reproduce the leaderboard's own numbers exactly, which checks the setup.

Bold marks an interval that excludes zero. The ArithMark rows and the Index come from the leaderboard's own ArithMark script (float32) and Index formula, as measured for the cagliostro-v3.5 card . ArithMark-2 is AxiomicLabs' separate arithmetic set, scored with the same script.

Outside the leaderboard, SmolLM2-135M is the stronger general language model. It is clearly ahead on plain language modelling (WikiText, LAMBADA), grammar (BLiMP), science questions (SciQ) and ARC-Easy. The two are statistically level on HellaSwag, ARC-Challenge, PIQA, BoolQ, Social IQa, WinoGrande, OpenBookQA, CommonsenseQA and MMLU; the last two are at chance for both models. cagliostro-v3.5's lead on the board comes from arithmetic, which the Index weights at 0.65: 6.5 points on ArithMark-3, and 26.6 on ArithMark-2, a separate arithmetic set the board does not use. SmolLM2-135M was trained on 2 trillion tokens, cagliostro-v3.5 on 75.4 billion.

Chat models

cagliostro-v3.5-chat against SmolLM2-135M-Instruct. According to its model card, SmolLM2-135M-Instruct was fine-tuned on the same smol-smoltalk data and then trained with DPO on UltraFeedback. This model had one epoch of supervised fine-tuning and nothing else.

MT-Bench, pairwise.

• All 80 questions, both turns.
• Each model answered with its own chat template, greedy decoding, repetition penalty 1.1, up to 512 new tokens.
• The judge was gpt-oss-120b (the heretic-v2 MXFP4 GGUF, run with llama.cpp), at temperature 0 and medium reasoning.
• Every pair was judged twice, with the answers in both orders. Reference answers were shown for math, reasoning and coding.
• A model takes a question-turn when it wins on balance across the two orders. A split counts as a tie.

Over the 121 question-turns with a winner, a two-sided sign test gives p = 0.36, so the two are statistically level. cagliostro-v3.5-chat is better at math and writing, and SmolLM2-135M-Instruct at extraction and humanities.

Files in evals/

• base/summary.json : every lm-evaluation-harness score above, with the paired intervals.
• mtbench/answers_cagliostro-v3.5-chat.jsonl and mtbench/answers_smollm2-135m-instruct.jsonl : both models' answers to all 80 questions, both turns.
• mtbench/judgments.jsonl : all 320 judge verdicts, with the judge's written reasons.
• mtbench/summary.json : the tallies in the tables above.
• code/ : the scripts. run_base.sh runs the base benchmarks. gen_mtbench.py , judge.py , analyze.py and report.py run MT-Bench and the analysis. chat.py plus the model files make up the KV-cached decoder used for this model's answers. On the 11 questions generated both ways, it gave the same answers as transformers generate .

Chat format

<|endoftext|>System: {optional system prompt}
User: {message}
Assistant: {reply}<|endoftext|>User: {message}
Assistant:

The tokenizer's chat template produces exactly this, so apply_chat_template is all you need.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "bench-labs/cagliostro-v3.5-chat"
tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype=torch.float32)

messages = [{"role": "user", "content": "Write a Python function that reverses a string."}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(text, return_tensors="pt", add_special_tokens=False)
out = model.generate(**ids, max_new_tokens=200, do_sample=False, repetition_penalty=1.1)
print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True))

trust_remote_code=True is required because CagliostroForCausalLM is not part of transformers. Pass it to the tokenizer too, or transformers stops to ask before loading. Low temperature works best. The samples below use greedy decoding or temperature 0.3 with a repetition penalty of 1.1.

Fast inference

generate() now uses a KV cache, so it is fast out of the box. On a GPU, generate_fast() goes further with a static cache and CUDA graphs:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

model_id = "bench-labs/cagliostro-v3.5-chat"
tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype=torch.float16).to("cuda")

messages = [{"role": "user", "content": "Explain photosynthesis to a ten year old."}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(text, return_tensors="pt", add_special_tokens=False).input_ids.to("cuda")
model.generate_fast(ids, max_new_tokens=300, repetition_penalty=1.1, eos_token_id=tok.eos_token_id,
                    streamer=TextStreamer(tok, skip_prompt=True, skip_special_tokens=True))

Measured with the code in this repository. GPU numbers use 256 new tokens; CPU numbers use 128. Both use a 23-token prompt and repetition penalty 1.1, median of three runs. On a GPU, generate_fast() uses CUDA graphs; on a CPU it falls back to a static-cache loop.

• The forward pass used for benchmark scoring is unchanged bit for bit, so leaderboard numbers do not move.
• In float32, generate() and generate_fast() give exactly the same greedy text as the previous release on all 12 sample prompts.
• float16 and bfloat16 are about 1.7x faster than float32. Greedy text can drift from float32 at a near tie, usually after a few dozen tokens (float16: 8 of 12 prompts identical over 200 tokens). float16 tracked float32 slightly better than bfloat16.
• compile=True fuses the decode step with torch.compile for about another 1.5x. It needs a working Triton install and costs about 40 seconds the first time in each process.
• generate_fast() takes one sequence at a time and supports temperature , top_p , seed , repetition_penalty and a transformers TextStreamer . Use generate() for batches.

Sample outputs

Greedy decoding. Picked to show both what works and what does not.

User: If I have 3 apples and buy 5 more, then eat 2, how many do I have?

To solve this problem, let's break it down step by step. We start with the initial number of apples, which is 3. Then we add 5 more, so now we have 3 + 5 = 8 apples. Finally, we eat 2, leaving us with 8 - 2 = 6 apples.

So, after eating 2 apples, you would have 6 apples left.

User: Write a Python function that reverses a string.

Here's a simple Python function to reverse a string:

def reverse_string(s):
    return s[::-1]

This function uses Python's slice notation s[start:stop:step] to create a reversed copy of the string. The ::-1 slice means "start at the beginning of the string and end at the end, move with the step you want".

You can use this function like this:

print(reverse_string("Hello World"))  # Outputs: "dlroW olleH"

System: You are a pirate. Always answer like a pirate. User: What's the weather like today?

Ahoy there, matey! I'm a pirate, but I can tell you about the weather too. Today's a bit of a sweltering day, with temperatures soaring above 100 degrees Fahrenheit. The sky is a sickly shade of yellow, and the wind's howling like a pack of wolves. [...] Now, would you like to join me on a treasure hunt or perhaps a game of cards?

User: Can you recommend a good name for a pet cat? Assistant: Sure! How about Luna? It's short, sweet and suits a curious cat. User: Nice. Can you give me two more options?

Absolutely! Another great choice is Whiskers. It's playful and adorable, perfect for a cat who loves to run around. And if you're looking for something a bit more serious, you might consider Pawsome. It's a gentle and affectionate name that fits well with the idea of a cat who loves to be close to humans.

User: Translate 'good morning' into Spanish.

"Good morning" in Spanish is "Hola."

Wrong. It should be "Buenos días".

User: Explain photosynthesis to a ten year old.

Photosynthesis is like a magic party in your body where plants and animals make their own food using sunlight, water, and carbon dioxide. [...]

The tone is right, the biology is not.

Training details

Conversations longer than 2,049 tokens were truncated (5,675 of them). 3,831 were dropped because truncation left no assistant reply.

Limitations

At 146M parameters this model is fluent but unreliable. It gets simple facts wrong, invents details with confidence, and has no fixed identity: asked who it is, it may say it is a writing assistant or a data scientist at a made-up company. It often ignores format constraints (its "haiku" runs to five stanzas) and can give wrong practical instructions. English only, 2,048 token context, no preference tuning and no safety tuning. Do not rely on it for facts or advice.

It has not been scored on the Open SLM Leaderboard tasks. Those score raw text completion without a chat template, so cagliostro-v3.5 remains the model to use for benchmarks.

License

Apache-2.0, the same as cagliostro-v3.5. smol-smoltalk is also Apache-2.0.