How big should a browser LLM's context be? Measure the KV cache, not the model

A model card says the context length is 262,144 tokens. You set n_ctx to something ambitious, the tab dies, and it is not obvious why: the model is only 2.6 GB and the machine has 24 GB of VRAM.

The reason is that llama.cpp reserves the entire KV cache when the model loads, not lazily as the conversation grows. So the question is never “how long a conversation do I want” — it is “how many bytes per token does this specific architecture cost, times the window I am allocating”.

That per-token figure varies by a factor of 64 across the models AgentOp ships. Any uniform default is therefore wrong for most of them.

The formula

For a standard attention model, the fp16 KV cache costs:

bytes/token = 2 (K and V) x layers x kv_heads x head_dim x 2 (fp16)

The parameter count barely appears. What matters is the number of layers, and crucially how many key/value heads the architecture keeps — which is exactly what Grouped-Query Attention (GQA) reduces.

What that costs in practice

Measured per model in the registry, with the allocation for an 8,192-token window:

Two entries deserve attention.

Phi-3.5-mini is the cautionary tale. It uses multi-head attention with no grouping: 32 key/value heads instead of 8. That is 768 KB per token, so an 8K window would reserve 6.3 GB of KV cache for a 2.4 GB model. Its default is capped at 2,048 tokens (about 1.5 GB) and its maximum offered window is 4,096. A 3.8B model is not “small” in the way that matters here.

The hybrids are almost free. Qwen 3.5 and LFM2.5 replace most layers with linear attention or short convolutions that carry constant-size state, keeping only a handful of full-attention layers. Qwen 3.5 4B costs 32 KB per token — a quarter of what Qwen 3 4B costs — so a long window on those models is cheap enough that it barely enters the budget.

Gemma 4 sits outside the table because its cost is sublinear: hybrid sliding-window attention (512/1024) with shared-KV global layers means 8K is inexpensive without a simple per-token constant.

Why 8192 became the default

The earlier default was a uniform 4,096 for everything, and it produced a specific, reproducible failure: the second query of a tool-calling session would overflow the window and error out.

That is not mysterious once you count what a tool-calling agent actually puts in the prompt — a system prompt, the JSON schemas for every tool, the conversation history, and the full text of each tool result. One document lookup can return a few thousand tokens on its own. The first exchange fits; the second does not.

So the default is now 8,192 for most models, chosen because weights plus KV plus compute buffers still fit inside a typical 8 GB GPU or 16 GB RAM budget, with the per-model exceptions above. Two adjustments sit on top:

• The in-page Context selector offers up to 32,768 for GQA models (16,384 for Qwen 3 0.6B, 4,096 for Phi-3.5-mini), for people who know their hardware.
• In Auto mode the client raises the default to 16,384 when inference is actually running on an allowlisted GPU vendor (NVIDIA or Apple).

That last one is deliberately vendor-gated even though the accelerator choice itself is probe-based. A passed GPU probe proves the device runs; it does not prove it has 16K-of-KV worth of headroom. Those are different claims, and conflating them turns a working session into an out-of-memory crash on someone else’s laptop.

The rule

Do not size a context window from the model card. Size it from 2 x layers x kv_heads x head_dim x 2 times the window you want, add the weights, add compute buffers, and check it against the smallest machine you intend to support — because in a browser tab, that machine is whoever opened the file.