Tool calling for small on-device LLMs that don't support tool calling

You attach a few tools to an AI agent, pick a small model that runs on your own machine, and press run. With a cloud model this is routine. With a model running in the browser, the agent refuses to start.

In AWFlow you can build agents: an LLM node with tools attached (search the page, call a sub-workflow, query a vector store). With OpenAI or Anthropic you pass tool schemas and get structured tool_calls back. With a model running in the browser (transformers.js or WebLLM), you don’t. Those runtimes give you text in and text out. LangChain’s tool-calling agents check for bindTools() and refuse local models outright:

Error: bind_tools() must be implemented

So I implemented it with prompt-based tool calling. Prompting the model is the easy part. The interesting part is parsing what it actually sends back, because every model family answers in the format it was trained on, whatever you asked for.

How the agent itself is wired (the graph, the loop, the pause) is covered in LangChain and LangGraph in the browser. This post zooms into one piece of it: tool calls on models that have no tool-calling API.

Describe the tools in the prompt

The tools are converted to OpenAI-style definitions (name, description, JSON Schema parameters) and rendered into a system-prompt section, one JSON object per line:

# Tools

You can call the following tools. Each is described as JSON (name, description, JSON Schema parameters):
{"name":"search_page","description":"Find text on the current page","parameters":{...}}
{"name":"save_row","description":"Append a row to the sheet","parameters":{...}}

To call a tool, reply with exactly this, one block per call, and nothing else:
<tool_call>
{"name": "<tool name>", "arguments": {<arguments matching the parameters>}}
</tool_call>

Only call the tools listed above. You will receive each result in a <tool_response> block.
When you have what you need, answer the user in plain text, without a tool call.

Earlier tool calls in the conversation are re-rendered in that same <tool_call> format, and results go back wrapped in <tool_response>. That way the transcript keeps showing the model the format you want.

I picked <tool_call> tags because many open models (Qwen-style chat templates, for one) were trained on something close to it. But “close” is the key word.

Accept what models actually write

Here’s what small models really return when asked for that format:

<tool_call>{"name": "search_page", "arguments": {"query": "price"}}</tool_call>   as asked
<function_call>{"name": "search_page", ...}</function_call>                      tag drift
[TOOL_CALLS] [{"name": "search_page", "arguments": {...}}]                         Mistral style
<function=search_page>{"query": "price"}</function>                                Llama style
```json
{"name": "search_page", "parameters": {"query": "price"}}
```                                                                                  fenced JSON
{"function": {"name": "search_page", "arguments": "{\"query\":\"price\"}"}}        nested + stringified args

…plus <think>…</think> reasoning blocks in front, and replies cut off at the token limit before the closing tag.

Rejecting all of that gives you an agent that “never calls tools”. Accepting all of it naively gives you an agent that runs things nobody offered it. So the parser is a list of patterns plus a strict filter:

const CALL_FORMATS = [
  // <tool_call>…</tool_call> / <function_call> / <tools>; closing tag optional (truncation)
  { pattern: /<(?:tool_call|function_call|tools)>\s*([\s\S]*?)\s*(?:<\/(?:tool_call|function_call|tools)>|$)/g },
  // [TOOL_CALLS] [ … ] — array runs to the end
  { pattern: /\[TOOL_CALLS\]\s*(\[[\s\S]*\])/g },
  // <function=name>{…}</function> — the name lives outside the JSON
  { pattern: /<function=([\w.-]+)>\s*([\s\S]*?)\s*(?:<\/function>|$)/g, nameGroup: 1 },
  // ```json {…} ```
  { pattern: /```(?:json|tool_call)?\s*([[{][\s\S]*?[\]}])\s*```/g },
];

Each match goes through one normaliser that accepts the usual spellings:

• name: name or function.name
• arguments: arguments, parameters, args or input, as an object or a JSON-encoded string
• a single call or an array of calls

function toRawCall(value: unknown, forcedName?: string) {
  if (!isPlainObject(value)) return undefined;
  const fn = isPlainObject(value.function) ? value.function : value;
  const name = forcedName ?? fn.name;
  if (typeof name !== 'string' || !name) return undefined;
  // With <function=name>{…}, the object IS the arguments.
  let args = forcedName ? value : (fn.arguments ?? fn.parameters ?? fn.args ?? fn.input ?? {});
  if (typeof args === 'string') args = tryParseJson(args);
  if (!isPlainObject(args)) return undefined;
  return { name, args };
}

Keep the leniency safe

A lenient parser is only useful if it can’t be talked into running things. Here is the path a reply takes, step by step.

Last resort: if nothing matched and the whole reply is JSON that parses as calls, accept that too. Some small models just answer {"name": …} with no wrapper.

Here is what that means for a few real-looking replies:

Why model-agnostic matters

My first version special-cased the model I was testing with. It worked great until the next model. A user can add chat models from the Hugging Face Hub to the extension (any with a browser-runnable ONNX build), so the parser can’t know the family. Matching on shapes instead of model names means any model that writes one of these formats works, with no per-family lookup table. Even the prompt itself is checked by a test to make sure it never names a model family.

It’s also a pure function: string in, { content, toolCalls } out. So every accepted format, malformed call and edge case has a unit test, with no model in the test run.

Does it actually work?

With 1–4B models, for agents with a few well-described tools: yes, well enough for things like “find the price on this page and save it to the sheet”. Where it breaks down:

• many tools with overlapping descriptions (the model picks the wrong one)
• long multi-step plans (it loses the thread after a few calls)
• deeply nested argument schemas

For those, the same agent node can switch to a cloud model or a bigger local one via Ollama (Ollama with workflow automation). Same workflow, different brain. If you are choosing an on-device model in the first place, how to choose, install and compare a local AI model walks through it.

If you’re doing this yourself:

If you’ve seen tool-call formats I haven’t covered, I’d like to see them in the comments.

The agents described here are part of AWFlow, a free browser extension for workflows and AI agents that runs in your browser.