Get started
// blog

Chat with a PDF without uploading it: local RAG in a browser tab

By Ozgun Ay · ·4 min read

ragembeddingstransformers.jsprivacypyodide

“Chat with your PDF” services almost all work the same way: you upload the file, it is parsed and embedded on a server, and your questions and the document text pass through someone else’s infrastructure. For a lot of documents that is fine. For a client contract, a medical letter or an unsigned commercial agreement, it is the reason the tool cannot be used at all.

The entire pipeline fits in a browser tab. This is how AgentOp’s Document Q&A agent does it, and what it costs.

The pipeline, stage by stage

1. Parse the PDF. pypdf runs in Pyodide — CPython compiled to WebAssembly — so the same library you would use server-side extracts the text client-side. micropip fetches the wheel from PyPI on first run and the browser caches it.

2. Chunk it. Overlapping character windows: 1,000 characters with 150 characters of overlap. The overlap exists so a sentence spanning a chunk boundary still appears intact in one of them; without it, a fact split across the seam is retrievable from neither side.

3. Embed the chunks. Xenova/all-MiniLM-L6-v2 through Transformers.js — about 25 MB, producing 384-dimensional vectors. This is the piece people expect to be impossible in a browser and it is the easiest part of the whole system. It is a small encoder, not a generative model.

4. Store them. Vectors are L2-normalised and persisted in IndexedDB, per agent, so re-opening the page does not re-embed the document. Memory-only is the fallback where IndexedDB is unavailable.

5. Retrieve. Brute-force cosine similarity over the normalised vectors, top-4 by default. Brute force sounds lazy and is correct here: a document produces hundreds or low thousands of chunks, and a dot product over a few thousand 384-dimensional vectors is microseconds. An approximate index would add a dependency, an index build step and a recall cliff to solve a problem that does not exist at this scale.

6. Generate. The retrieved passages plus the question go to whichever model the agent is running on. Locally that is llama.cpp in WebAssembly with WebGPU offload; the same agent can also run on OpenAI or Anthropic with the user’s own key.

The tunable parameters are exactly three:

RAG_CHUNK_SIZE = 1000
RAG_CHUNK_OVERLAP = 150
RAG_DEFAULT_TOP_K = 4

The part that surprises people

Retrieval does most of the work, and retrieval does not need a big model.

A small local model is bad at knowing things and decent at reading what is placed in front of it. That is precisely the shape of a RAG task: the retriever finds the four relevant passages, and the model’s job is to answer from them rather than from its parameters. So a 4B model that would embarrass itself on open-ended trivia does respectably on “what is the notice period in this contract”.

The corollary is the honest limit. Document Q&A works well; a conversation about the document across many turns degrades, because that starts to depend on the model rather than the retrieval.

What it costs

On an NVIDIA RTX 4090 in Chrome, with Qwen 3 4B (a 2.6 GB one-time download, cached by the browser afterwards), generation runs at about 57 tokens/second.

The one-time costs on first run are the model, the Pyodide runtime, the pypdf wheel and the 25 MB embedding model. Everything is cached afterwards, and a downloaded standalone agent works with the network switched off.

The ongoing cost is zero, because the compute is the user’s.

The limits, stated plainly

  • You need a GPU. WebGPU with a graphics card on Chrome or Edge, or an Apple Silicon Mac. Without one it falls back to CPU, which works and is slow.
  • Not phones. WebGPU plus a multi-gigabyte download is not a phone experience and we do not pretend otherwise.
  • Sub-2B models fail multi-turn memory. They fail a “what did I tell you my name was” test most of the time; 3B and above mostly pass.
  • Retrieval quality is a chunking problem. A table split across chunk boundaries retrieves badly, and no amount of model quality fixes it.

Why the browser rather than a desktop app

If you want a local document assistant for yourself, install Ollama or LM Studio. They are faster, they run bigger models, and they are the right tool.

The browser version exists for the other person. An agent exports as a single HTML file: the recipient double-clicks it, the model downloads once, and it runs on their machine. That is the difference between “open this” and “install this, pick a quantisation, paste this system prompt” — and it is the only reason to accept the browser’s constraints.

There is a second, quieter benefit. When the document never leaves the tab, there is no data-processing agreement to negotiate, no retention policy to explain and no breach surface to worry about. The privacy claim is architectural rather than a promise, which is a different kind of claim to make to a lawyer or an accountant.

You can try Document Q&A in the browser without signing up, or compare the ways to run a model locally if you are still choosing an approach.

More engineering notes

Reading receipt photos with a 40 MB OCR model in the browser TrOCR-small is a line recognizer, not a page reader. Handed a whole receipt photo it returns roughly one line… Tool calling on small local models: native templates vs GBNF grammar Half the model families we ship cannot use llama.cpp's native tool calling, each for a different reason. A GB… How big should a browser LLM's context be? Measure the KV cache, not the model llama.cpp reserves the whole KV cache up front, so a model's advertised 262K context is irrelevant in a brows…

AgentOp turns this into something you can hand to someone else: an AI agent exported as a single HTML file that runs a local model on their machine, with no install and no server. Try one in your browser or compare the ways to run a model locally.