ajvikram / toolcall-2b

Toolcall-2B

A 2B function-calling model for local agent tool routing, fine-tuned from Qwen/Qwen3.5-2B and measured on the Berkeley Function Calling Leaderboard v4 harness.

It is built for the decision an agent makes hundreds of times per task: which tool to call, with which arguments, in what order, and whether to call one at all. Those decisions are cheap in judgment and expensive in tokens when they run through a frontier model.

Results

Measured with the BFCL v4 harness on one server (vLLM, bf16, temperature 0.001, 128k serving context), identical settings for every model. The base model was re-scored under the same conditions rather than quoting published numbers, so these are like-for-like.

* The free search-API tier allows 250 queries a month and one pass over this category costs about 165, so this figure is not reliable and the 22-category overall understates the model. The 20-category mean is the honest comparison.

Largest per-category gains over the base:

All four multi-turn categories are above the base: base 27.0 → 31.0, long context 24.0 → 30.5, missing parameter 14.0 → 18.0, missing function 6.0 → 9.0.

Known weakness: the refusal family sits below the base — irrelevance 77.9 → 72.5, live_relevance 87.5 → 81.2. A later round that added more refusal demonstrations fixed part of this but cost more in multi-turn and memory than it gained, so it was not shipped.

Quantized builds

GGUF builds for llama.cpp and Ollama are at ajvikram/toolcall-2b-gguf : Q4_K_M (1.22 GB) runs on a laptop CPU at roughly 33 tokens per second, with Q5_K_M, Q8_0 and f16 also available.

Usage

The model keeps Qwen3.5's native tool-call format, so it drops into any stack that already parses Qwen3.5 tool calls.

vllm serve ajvikram/toolcall-2b --dtype bfloat16 --max-model-len 131072 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string", "description": "City name"}},
            "required": ["city"],
        },
    },
}]
r = client.chat.completions.create(
    model="ajvikram/toolcall-2b",
    messages=[{"role": "user", "content": "What's the weather in Berlin?"}],
    tools=tools,
    temperature=0.0,
)
print(r.choices[0].message.tool_calls)

Emitted calls look like this, which is the base model's own format:

<tool_call>
<function=get_weather>
<parameter=city>
Berlin
</parameter>
</function>
</tool_call>

Thinking is off by default, matching how the model was trained and evaluated.

Training

Supervised fine-tuning, 16-bit LoRA (rank 16, alpha 32, all linear projections), loss on assistant turns only, sequences to 8,192 tokens, learning rate 3e-5, 0.6 epoch, batch 4 with 8-step accumulation. One run of 3.2 hours on a rented RTX A6000; final training loss 0.330, validation loss 0.247.

Prompts are rendered with the model's own chat template at training time and at evaluation time, so the two never diverge.

Data

61,712 examples. Every public source is permissively licensed; a CC-BY-NC set was excluded on purpose so these weights can be Apache-2.0.

Two augmentations, applied deterministically: function masking, which renames tools and parameters to opaque identifiers consistently across the schema and the calls so the model reads descriptions rather than memorising names; and distractor tools drawn from other examples, so tool selection is exercised.

The ingredient that mattered

Four earlier rounds on public data alone raised single-turn accuracy a great deal and made multi-turn worse than the base every time. Public corpora teach the shape of a tool call, not the habit of working a stateful task to completion.

Round five added 2,134 conversations generated against six simulated stateful APIs — a file system, task tracker, mailbox, key-value memory, calendar and shop — where a larger model played the assistant, a second model played a user who revealed the task gradually, and every episode was kept only if a checker confirmed the final state was correct. Episodes were rejected when the goal was not met, when a tool errored more than once, when an unrelated tool was called, or when the model guessed instead of asking for information it had not been given. That single change moved every stateful category above the base at once.

Evaluation, reproduced

# harness: fork of ShishirPatil/gorilla with a handler for Qwen3.5's XML tool-call format
bfcl generate --model ajvikram/toolcall-2b-FC --test-category all_scoring \
  --skip-server-setup --local-model-path <merged-dir> --num-threads 16
bfcl evaluate --model ajvikram/toolcall-2b-FC --test-category all_scoring

Limitations

• Refusal behaviour is below the base model: it is more willing to attempt a request no available tool can serve.
• The web-search score is not reliable, for the quota reason above.
• Numbers here come from one person's harness runs, not from the official leaderboard. They are reproducible with the commands above, and the base model measured on the same rig is the comparison that matters.
• English only, and evaluated only on this benchmark's tool-calling tasks.

Citation

Built on Qwen3.5-2B; evaluated with the Berkeley Function Calling Leaderboard harness. Training data credits are listed above, and the episode generation follows the verified-trajectory approach described in APIGen-MT .