Engineering notes
How AgentOp actually works, written down while building it: WebAssembly limits, WebGPU behaviour, tool calling on small models, and what on-device inference can and cannot do. Numbers are measured, and the hardware is named.
-
Chat with a PDF without uploading it: local RAG in a browser tab
The whole retrieval pipeline — PDF parsing, chunking, embeddings, vector search and generation — fits in a browser tab with a 25 MB embedding model. Here is how each stage works and where the honest limits are.
-
Reading receipt photos with a 40 MB OCR model in the browser
TrOCR-small is a line recognizer, not a page reader. Handed a whole receipt photo it returns roughly one line. Getting a usable expense table out of it needed row-projection segmentation, an Otsu threshold, and a rule that the model never produces a number.
-
Tool calling on small local models: native templates vs GBNF grammar
Half the model families we ship cannot use llama.cpp's native tool calling, each for a different reason. A GBNF grammar that constrains sampling to valid JSON works for all of them. Here is the matrix and the failure modes.
-
How big should a browser LLM's context be? Measure the KV cache, not the model
llama.cpp reserves the whole KV cache up front, so a model's advertised 262K context is irrelevant in a browser tab. The number that matters is KV bytes per token, and it varies 64x across the models we ship.
-
Why models over 4 GB failed in the browser, and the two fixes
A 5 GB GGUF would not load in a browser tab, and the reason turned out to be two separate 4 GB ceilings: WebAssembly's 32-bit address space and the JavaScript heap. Here is how each one was fixed.