// blog

Reading receipt photos with a 40 MB OCR model in the browser

By Ozgun Ay · 7 September 2026 ·5 min read

ocrtransformers.jsvisionreceipts

The Receipt & Invoice Extractor is an AgentOp agent that turns a photographed receipt into an expense table you can export as CSV, entirely in the browser. The OCR model is Xenova/trocr-small-printed — about 40 MB, running through Transformers.js.

Two things about it were not obvious until real photographs went through, and both changed the design.

TrOCR-small is a line recognizer

The capability was verified early against a single-line test image, which passed, and that hid the actual behaviour. Handed a full receipt photo, TrOCR-small returns roughly one line of text for the whole page. It is a line-level recognizer; the page is not its unit of work.

So the agent has to do the page layout itself: find the horizontal bands of text, crop each one, and call OCR once per strip. Warm, each strip takes about 0.7 seconds.

The naive version of that — threshold the image against its mean brightness, find rows with ink, cut there — does not survive contact with a phone camera. Three specific choices were forced by real photographs, measured against receipt photos from Wikimedia Commons.

Ink is judged against a local neighbourhood, not the page

A phone photo of a receipt has a lighting gradient across it. One side is brighter than the other, often by more than the difference between paper and print. Threshold against the page mean and the shadowed half is uniformly classified as ink, at which point the whole receipt collapses into a single band.

The fix is to compare each pixel against a local horizontal neighbourhood — a 25-pixel window, with ink defined as being at least 14 grey levels darker than that window’s mean:

const MAX_LINES = 28;     // OCR budget per receipt (~0.7 s per line warm)
const LOCAL_WINDOW = 25;  // width of the local-contrast neighbourhood, px
const LOCAL_K = 14;       // how much darker than its neighbourhood ink must be

A running prefix sum keeps that cheap: the neighbourhood mean for every pixel is two array lookups and a subtraction.

The text/gap split is an Otsu threshold, not a constant

Having built a per-row ink profile, you still need to decide which rows are text and which are the gaps between lines. A fixed fraction of the page width is the obvious approach and it is wrong, because the right fraction depends on the photo: exposure, resolution, how tightly the receipt is printed.

Otsu’s method computes the threshold per photograph by finding the split that minimises intra-class variance in the row-ink profile. It adapts on its own, which is exactly what a fixed constant cannot do.

Bands far taller than their neighbours get re-cut

Crumpled paper makes adjacent lines touch. Where they touch, segmentation produces one band roughly twice the height of the others, containing two lines that TrOCR will render as one garbled string. Any band far taller than the local median is split evenly into the number of lines its height implies.

Before these three changes a real receipt segmented into 1–2 bands and the OCR text was unusable. After them, the same photograph yields 7–28 lines.

When a receipt exceeds the MAX_LINES budget the agent keeps the first and last halves rather than truncating: the vendor is at the top and the total is at the bottom, and a naive head-truncation drops the total every single time.

Digital PDFs skip all of this. They go through pypdf and come out exact and instant — segmentation is only for photographs.

The model never produces a number

This is the rule the whole extractor is built around, and the browser pass proved why.

Totals, tax, dates and currency are extracted by deterministic regexes in Python, not by the language model. The LLM is used only for the vendor name and the spending category, and every failure there degrades to a deterministic guess. The extractor also records a total_source field, so the interface can flag a total that was merely the largest amount found on the page rather than a value read from a line labelled “total”.

The amount regex refuses bare integers. Money must have a two-decimal fraction, thousands grouping, or an adjacent currency symbol:

_AMOUNT_RE = re.compile(
    r"(?<![0-9])"
    r"(?:"
    r"[$€£₺¥]\s?[0-9]{1,6}(?![0-9.,])"  # symbol + whole units (JPY)
    r"|[0-9]{1,3}(?:[.,\s][0-9]{3})+(?:[.,][0-9]{2})?"      # grouped thousands
    r"|[0-9]{1,6}[.,][0-9]{2}"                              # plain decimal amount
    r")"
    r"(?![0-9])"
)

That restriction is not theoretical. Receipts are dense with long digit strings that are not money, and before the rule existed the extractor reported a UPC (04900005375) and a product code (#60101) as receipt totals. A bare integer on a receipt is more often a barcode than a price.

Questions about the receipt are answered over the extracted table — a facts sheet — not over a vector index. That is why this template declares the vision capability alone and pulls in no embedding engine at all.

What it is actually good for, measured

Honest results from the browser verification pass:

Digital PDF invoices: exact. pypdf, no OCR, instant.
Photographs: partially legible per line. Vendor names and item/price lines usually survive — a real extracted line reads B&G101LPAPERTONS/DISH/110&0 10.99. The totals block often does not.
Non-Latin receipts: noise. A Japanese MaxValu receipt came back unusable. The model is English printed text only.

That gap is precisely why the deterministic scanner flags a guessed total instead of asserting one.

Reading whole photographed pages reliably needs a page-level vision-language model; Florence-2-base is the standing candidate. The line-strip approach here is what the shipped 40 MB pin can do, and it is enough to turn a pile of receipts into a table you can check and correct — which was the actual job.

One prod-only trap

Transformers.js resolves an image argument by fetching it, and RawImage.fromURL() runs fetch() even on a data: URL. fetch is governed by connect-src, so every agentop_ml.ocr(dataURL) call died with TypeError: Failed to fetch under the production Content Security Policy until data: was added to connect-src.

Local testing runs with DEBUG=True and no CSP, so this class of bug never appears in development. The earlier vision verification used an https: image URL, which is why it did not surface then either.