Tool calling on small local models: native templates vs GBNF grammar

An agent is a model plus tools. Getting a 4B model running in a browser tab to reliably emit a parseable tool call was the single largest time sink in building AgentOp, and the reason is that “supports tool calling” means at least three different things depending on the family.

This is the matrix across the 10 families in the registry, and the failure modes worth knowing before you hit them.

Two paths

Native. Pass tools and tool_choice to createChatCompletion and let llama.cpp render them through the model’s own chat template, then parse the structured tool_calls off the response. Clean, and it only works when the chat template is genuinely tool-aware.

Grammar. Generate a GBNF grammar from the tools’ JSON schemas and constrain sampling with it, so the model can only emit either a valid tool call or a plain answer. This works with any instruction-following model, because it operates at the token-sampling level rather than on the output text. There is no free-form parsing step to get wrong.

AgentOp tries native first where the family supports it, and falls back to grammar whenever a native attempt fails mid-turn.

Which families need which

Five of ten need the grammar path, and each one fails differently, which is the point: there is no single “non-compliant” behaviour to special-case.

Gemma has no tool section in its template at all, so llama.cpp’s generic handler takes over and the model prints the call as plain text. The observed output was literally get_data_summary{} in the middle of a sentence — not JSON, not a tool call, just prose that happens to look like one.

DeepSeek R1 has two problems at once. The distill templates are not tool-aware, and the models emit <think>…</think> blocks before answering, which breaks a structured parse even when a call is in there somewhere. The client strips think blocks and raises the generation budget so answers are not truncated mid-reasoning.

Granite, LFM2.5 and Ling each have a real native tool format — just not the JSON one the client parses. Granite uses XML-ish tags, LFM2.5 emits Python call syntax, Ling nests <arg_key>/<arg_value> pairs. Supporting three bespoke parsers to recover the same information a grammar can impose directly is not a good trade.

One more shared detail: Granite, LFM2.5 and Ling all default enable_thinking=True in their chat templates. The client always sends chat_template_kwargs: {enable_thinking: false}, so none of them reaches the parser with think blocks, and all three are marked reasoning: False like Qwen 3.

Four failure modes worth designing for

These cost real debugging time and none of them is obvious in advance.

Models hallucinate arguments for zero-parameter tools. Ask for get_time() and a small model will confidently pass it a timezone it invented. Retry once with no arguments before failing the call; it almost always succeeds.

A tool that throws should return its error to the model. The tempting design is to catch the exception and substitute a canned string. Then the model never learns what went wrong and cannot recover. Returning {error: "..."} gives it one chance to explain or try something else, and the synthesis pass turns that into a real sentence for the user.

Do not re-run the loop when a follow-up completion fails after a tool has already executed. Retrying the whole turn re-executes the tool, so any side effect happens twice. Synthesise an answer from the tool result you already have instead.

Grammar sampling can hit a null function pointer under GPU offload. In the shipped WebAssembly build this shows up intermittently, and the right response is to retry once unconstrained rather than abandoning the GPU: dropping the grammar for that session degrades reliability slightly, while falling back to CPU degrades speed by an order of magnitude.

Worth encoding, worth skipping: enum constraints belong in the grammar as literal alternatives, because that is where a small model most often goes off-script. Free-string properties are not worth constraining beyond “is a JSON string” — the grammar gets large and the model was not going to violate that anyway.

How it is tested

Assertions are shared between the run page’s eval panel and a CLI that runs a YAML scenario suite across the model matrix in a real Chromium with WebGPU. The suite covers a greeting smoke test, one poke per declared tool, a name-recall memory case and a seeded-fact grounding case.

The memory case earns its place: “what did I tell you my name was” fails on most sub-2B models and passes on most 3B and above. One line of YAML, and it catches the class of regression where an agent looks fine in a single-turn demo and falls apart on the second question.