OCR is a three-stage pipeline: detect the boxes, recognise the characters, then lay them out. Classical engines run the stages as separate models and accumulate their errors; Donut and the vision-language models merge them into one network that trades data for structure. Learn the stages — and CTC, the alignment trick underneath them — and you can read, choose and debug every OCR stack in production.
Layer 1 turns pixels into characters; layer 2 labels and orders the regions; layer 3 produces { invoice_total: 1,463.20 }. Each rung inherits the errors of the one below it: a 3,000-character page at 2% CER is 60 wrong characters, and 1,000 invoices × 4 fields at 98% field F1 is 80 wrong values a day.
3,000 chars ≈ 50 lines ≈ 500 words · 2% CER = 60 wrong charactersDB and CRAFT score every pixel and group the positives into quads — detection is scored by IoU, where a box missing a fifth of a 100×20 line still scores 0.80. Each crop is warped to a fixed height, the width becomes the time axis (320 px ÷ 4 = 80 steps), and a CRNN + CTC reads the characters. Donut merges all three stages into one ViT encoder + decoder.
IoU 0.923 at a 4 px shift · 0.80 when 20% of the line is missingOne distribution per time step over vocabulary + blank; the loss sums over every alignment that collapses to the target — C(T+S, 2S) = 462 paths for “cat” over 8 steps. The collapse rule is merge repeats first, then delete blanks: a a ∅ a → “aa”. Reverse the order and you get “a”. Greedy decoding stays within ~1% CER of beam search on clean print.
a a ∅ a → “aa” · l ∅ l → “ll” · l l l → “l”OCR is a three-stage pipeline — detect text boxes, recognise the characters, then lay them out — and every modern system either reorders those stages or merges them: Donut folds all three into one image-in/JSON-out model, a vision-language model adds reasoning to the same job, and CTC is the alignment trick that makes the recognition stage trainable without character timings.
By the end you will be able to read a detection metric the way the text sees it (a 0.5 IoU gate lets a box miss half a line; 0.80 still misses a fifth); implement the CTC collapse rule in the right order and count the alignments it sums over (462 for “cat” over 8 steps); explain why a 32×80 line crop gives 20 time steps and why T ≥ S is a hard constraint; compute CER and WER on a 3,000-character page and field F1 across 4,000 field values; describe what layout parsing adds and why a 6 × 20 table is 120 cells of structure rather than 120 strings; and pick between Tesseract, EasyOCR, PaddleOCR, TrOCR, Donut and a VLM by language coverage, latency, accuracy and cost.
Receipts, invoices, IDs, scanned books, forms, whiteboards, signs, screenshots. Getting the characters out is the easy half; knowing which one is the total is the half that pays. The field splits into three layers, each with its own tools, metrics and failure modes.
Take a single scanned page. It is about 3,000 characters — roughly 50 lines of 60 characters, or 500 words. Now ask three different questions of it, and notice that each one needs a different machine:
Here is the arithmetic that makes the ladder feel real. At a healthy 2% character error rate, a 3,000-character page still has 60 wrong characters — one typo every 100 characters, or every other line. Word error rate is worse, not better: a single wrong character inside a five-letter word fails the whole word, so 60 character errors can produce up to 60 wrong words out of 500 — up to 12% WER from 2% CER. Every layer above OCR inherits that noise.
Now price the last layer. If a form has four fields you care about and you process 1,000 invoices a day, that is 4,000 field values a day. At 98% field F1, 80 values a day are wrong — 80 invoices someone has to fix. At 99.5% F1 it is 20. The two percentage points between those numbers are worth more than the whole OCR stage, and they are decided by the layer above it.
A scanned page holds 3,000 characters. Your pipeline reports 1% character error rate. Roughly how many wrong characters are on the page — and why is the word-level rate likely higher?
The rest of this lesson walks the ladder from the bottom, because that is the order in which the systems are built: find the text, read the text, then put the page back together. The last chapter is about choices — six production tools that each live at a different rung.
The classical OCR pipeline is five stages and three neural networks: detection finds boxes, recognition reads each crop, layout rebuilds the page. Every modern system either merges those stages into one model or reorders them — and the trade is error accumulation against data appetite.
The pipeline is worth knowing by heart, because the box coordinates it produces are the only bridge between pixels and everything above:
Two design decisions inside that diagram drive everything else. First, the recogniser wants a fixed height — you resize every crop to 32 or 48 pixels tall and let the width carry the sequence. Second, the width becomes the time axis for CTC: a 320 px-wide line crop that the CNN downsamples 4× gives 80 time steps for the characters in the line. Chapter 04 is entirely about why that works.
The cost of the design is error accumulation, and it compounds faster than people expect. Suppose detection finds 99% of lines, recognition reads 98% of characters correctly, and the layout stage orders 97% of regions correctly — each number looks fine:
End-to-end models make a different bet: one network, one training signal, no hand-off. Donut (2022) is a ViT encoder plus a text decoder that reads the whole page image and emits the answer string — often the JSON you wanted, with no detector, no recogniser and no layout module in sight. TrOCR does the same trick at line level. Qwen-VL-OCR and the other vision-language models do it for whole pages, in dozens of languages, with reasoning on top. The price is data: Donut wants hundreds to thousands of labelled examples of your document type, where the classical pipeline starts working on day zero.
The counter-intuitive consequence is a deployment split. For high-volume printed text, the classical pipeline wins on latency and cost: a 3,000-character page costs roughly 150 ms (one detection pass, 50 line crops batched through the recogniser, a layout pass). A VLM that has to decode 3,000 output characters — about 1,200 tokens — takes seconds, because generation is sequential. For structure and reasoning, the VLM is the only one that even plays.
# the classical pipeline, production packaging included
result = PaddleOCR(lang="en").ocr("page.png") # boxes + text + scores
# the end-to-end bet: an image in, JSON out — no detector anywhere
processor = DonutProcessor.from_pretrained("naver-clova-ix/donut-base-finetuned-cord-v2")
model = VisionEncoderDecoderModel.from_pretrained("naver-clova-ix/donut-base-finetuned-cord-v2")
# the 2026 default for hard pages: ask a vision-language model
# "transcribe this page, then output the line items as JSON"Detection is 99% accurate, recognition 98%, layout 97%. What is the end-to-end success rate if the three stages are independent — and what is the lesson?
Six ways to read a page. Flip the metric between latency, language coverage, error and cost, then pick one engine to see why you would reach for it — and when you would not.
Latency and error bands are teaching ranges for clean printed English, not a benchmark run: your scans, your language and your hardware move them. The cost bars are exact arithmetic on the latency and the rate you choose.
Read the comparator as a decision, not a leaderboard: slide the GPU rate and the cost bars move, but the shape does not. The classical pipeline buys throughput; the end-to-end model buys structure; the VLM buys understanding. Chapter 07 turns that into tool choices per task.
Text detection takes an image and returns quadrilaterals — one per word or line — with a confidence score. Two families do it: regression, which predicts the box coordinates directly, and segmentation, which predicts a per-pixel map and then groups pixels into boxes. The second family won.
Regression detectors (EAST, TextBoxes++) treat the problem like object detection: from a feature map, predict a box per anchor point, plus an angle. Text is different from a cat, though — the boxes are long, thin, often rotated, and packed close together, so generic object-detection recipes struggle with the aspect ratio.
Segmentation detectors reframe the problem: predict one number per pixel — “am I inside a text region?” — then group the positive pixels into boxes. DB(Differentiable Binarization, 2020) predicts a probability map and a learned threshold map, combines them inside the network so the binarisation step is differentiable, and trains on shrunken text regions. CRAFT predicts two maps instead: a region score for each character and an affinity score for the space between characters, which is why it handles curved and rotated scene text so well.
Either way, the output is not text. It is pixels scored, then grouped:
import cv2
import numpy as np
prob = model(image) # DB or CRAFT: a map, same H×W as the input
binary = (prob > 0.3).astype(np.uint8) # DB learns this threshold; 0.3 is the paper's default
contours, _ = cv2.findContours(binary, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
boxes = []
for contour in contours:
if cv2.contourArea(contour) < 12: # drop specks — noise becomes boxes otherwise
continue
quad = cv2.boxPoints(cv2.minAreaRect(contour))
boxes.append(quad) # a rotated quad, not an axis-aligned rect
print(f"{len(boxes)} candidate boxes from one page")Now quality. Detection is scored with IoU — the area of the overlap divided by the area of the union — and the arithmetic is easy enough to do in your head, which is why it is worth memorising a few landmarks. Take a ground-truth line of 100 × 20 pixels:
At IoU 0.5 a predicted box can be half the line and still count as a correct detection. At 0.80 — the fourth row above — the box misses a fifth of the characters, and the recogniser will never see them. This is why text detection reports recall of lines alongside IoU: a missed line is a lost sentence, while a sloppy box is only a few lost characters. And it is why pipelines run non-maximum suppression after detection: segmentation maps produce several candidate boxes per line, and NMS keeps the highest-scoring one, discarding neighbours whose IoU with it exceeds a threshold (0.5 is the usual value).
A detector never returns text — it returns boxes and scores. Slide the score threshold to drop weak boxes, turn NMS on to remove duplicate predictions of the same line, and select a box to see its IoU term by term. Click a box on the page or use the selector.
A 0.5 IoU gate is loose for text: the box at 0.80 still misses a fifth of its line, and every character in the missing part is lost — a detection metric can look healthy while the text quietly loses words. Watch the table rows (score 0.44 and 0.38) disappear when you raise the threshold past them.
One more job lives at this stage: reading order. Once boxes exist, the simplest correct-enough rule is to sort them into horizontal bands, then left to right inside each band — which is what the board’s reading-order overlay draws. It works for a single-column Latin page and fails the moment there are two columns, which is the problem chapter 05 hands to a layout model.
The recogniser sees a fixed-height crop and must produce a character sequence of any length. Nothing in the labels says which frame holds which letter. CTC solves that mismatch with one extra symbol and one collapse rule — and it is the reason a CRNN trained in 2015 still reads your receipts.
Watch the shapes. A line crop of 320 × 32 pixels goes through a CNN whose four pools take the height from 32 to 2 and the width from 320 to 80, then a mean over the two remaining rows drops the height — giving 80 feature vectors, 80 time steps. The BiLSTM reads those 80 vectors in both directions, and a linear head emits a probability distribution over the vocabulary at every step. The target is just "hello": five characters, no timings. The model must learn the alignment by itself.
CTC (Connectionist Temporal Classification, Graves et al., 2006) does it by adding one symbol to the vocabulary: the blank, written ∅ here. At every time step the model distributes probability over vocabulary + blank. A path is one choice per step; a target is what the path becomes after two operations:
The blank has two jobs, and both are easy to underestimate. The first is separator: l ∅ l is “ll” while l l is “l”, so without a blank no model could ever write a double letter. The second is silence: a step that says “nothing here” lets one model handle a 5-character line in 12 steps and a 40-character line in 80 — the blank absorbs the mismatch between frame count and character count.
The loss is the interesting part. You cannot ask “did step 3 predict the letter e?” because nobody labelled the steps. So CTC does not pick an alignment:
That summation is the whole trick: every alignment that spells the right answer gets credit, so the model is free to put the e one frame early or one frame late, and gradients still flow to the right output. Now you can read the output of a trained model.
Every time step is a probability distribution over the vocabulary plus the blank. Greedy decoding takes the argmax at each step, then applies the collapse rule: merge runs of the same symbol, then delete blanks. Slide the cursor to build the path; the toggle shows the collapse done in the wrong order.
This target has an adjacent repeat, so the C(T+S, 2S) formula is an upper bound — the true number of valid alignments is smaller, because the two equal letters must be separated by a blank.
Greedy vs beam. The lab decodes greedily: take the argmax at each step, collapse. A beam search keeps the top-k prefixes (k = 5 is typical) and merges equivalent prefixes, which can recover a target whose best single path is wrong. On clean printed text the model is confident at every step, greedy’s argmax is usually the beam’s winner, and the two are within about 1% CER of each other; beam costs k× the decode time and buys almost nothing. On noisy scans, handwriting or ambiguous glyphs, the per-step argmax starts to lie and beam earns its keep — benchmark both on your documents before paying for it.
The other decoder. Not every recogniser uses CTC. TrOCR and Donut use a transformer decoder: it attends over the encoder’s features and its own previous outputs, and emits characters or whole structured strings directly — no blank, no collapse, no monotonic left-to-right constraint. That freedom is why end-to-end document models can emit JSON (a brace, a key, a value, nested objects) instead of a line of text. It is also why they can hallucinate: an attention decoder with no alignment constraint will happily produce a plausible word that is not on the page, where a CTC model fails loudly by dropping or repeating characters.
import torch
import torch.nn.functional as F
BLANK = 0 # the convention: index 0 is the blank, characters start at 1
def ctc_loss(log_probs, targets, input_lengths, target_lengths):
"""
log_probs: (T, N, C) log-softmax over vocab + blank
targets: (N, S) character ids, no blanks
input_lengths: (N,) time steps actually used
target_lengths: (N,) target length
"""
return F.ctc_loss(log_probs, targets, input_lengths, target_lengths,
blank=BLANK, reduction="mean", zero_infinity=True)
def greedy_ctc_decode(log_probs):
preds = log_probs.argmax(dim=-1).transpose(0, 1).cpu().tolist()
out = []
for seq in preds:
decoded, prev = [], None
for index in seq:
if index != prev and index != BLANK: # merge repeats, drop blanks
decoded.append(index)
prev = index
out.append(decoded)
return outA model emits the greedy path a a ∅ a b b ∅ c over 8 time steps. What text does it produce — and what would the wrong collapse order (blanks first) give?
Recognition gives you strings in boxes. Layout parsing decides which boxes are a title, a paragraph, a table or a footnote, and in what order a human would read them. Document understanding then extracts the fields — and the conversion to plain text, Markdown or JSON is where most pipelines quietly lose the structure they worked for.
Reading order looks trivial until you leave the single-column Latin page. A newspaper spreads one story across two columns; Arabic runs right to left; Japanese can run top-to-bottom and right-to-left; forms interleave labels, boxes and footnotes. Sorting by y and then x reads a two-column article across the columns, producing two interleaved half-sentences — and every model above inherits the garbage:
Tables are the sharpest test of this layer. A 6 × 20 table is 120 cells. Plain OCR returns those cells as a stream in reading order and the row/column relationships vanish. A structure model recovers cells, rows, columns and spans; at 95% cell-level accuracy six cells are still wrong per table — and if one of them is in the total row, the extracted number is wrong even though the OCR was excellent. Tables are scored with cell-level F1 and with TEDS (tree-edit-distance similarity, introduced with the PubTabNet table benchmark), which compares the structure, not just the characters. Donut scores its JSON output the same way — a normalised tree edit distance over the output tree.
Key-value extraction takes the layer one step further. Two architectures dominate: LayoutLMv3 takes the image plus the detected text plus each token’s position, so the model knows that “1,463.20” sits to the right of “Total” at the bottom of the page; Donut skips text detection entirely and reads the image to JSON in one pass, which wins on visually rich documents (receipts, forms) and needs a few hundred labelled examples to fine-tune. Then the fields are scored the way fields are scored everywhere:
Nineteen recognised boxes from one invoice. The reading order decides whether the summary block lands between the item rows; the format decides whether the table survives at all. Flip both and watch the same boxes become a paragraph, a table, or a schema.
NORTHWIND TRADING Invoice 2026-0417 Issued 2026-09-15 Item Qty Price Consulting hours 12 840.00 Subtotal 1,240.00 Platform licence 1 250.00 Tax (18%) 223.20 Data migration 3 150.00 Total 1,463.20 Due 2026-10-15 Page 1 of 1
A real pipeline makes this decision once, at the top of the project: plain text is for search and accessibility, Markdown is for humans, JSON is for the next program. Everything downstream inherits the choice — and a naive y-then-x reading order destroys the table before the formatter ever runs.
The same page reports 2% CER and 5% WER. Both are correct. Why is the word-level number larger?
The source’s build is three pieces: a tiny CRNN that turns a line image into a sequence of distributions, a synthetic dataset that needs no downloads, and the CTC loss that ties them together. Write it once and the production libraries stop being magic.
The recogniser first. Every choice in it has a reason: the CNN’s four pools cut the crop’s height from 32 rows to 2 and its width by 4, the mean over those two rows leaves a short sequence along the width, the BiLSTM reads that sequence in both directions, and log_softmax produces a proper distribution per time step — the input CTC expects.
import torch
import torch.nn as nn
import torch.nn.functional as F
class TinyCRNN(nn.Module):
def __init__(self, vocab_size=40, hidden=128, feat=32):
super().__init__()
self.cnn = nn.Sequential(
nn.Conv2d(1, feat, 3, 1, 1), nn.BatchNorm2d(feat), nn.ReLU(inplace=True),
nn.MaxPool2d(2),
nn.Conv2d(feat, feat * 2, 3, 1, 1), nn.BatchNorm2d(feat * 2), nn.ReLU(inplace=True),
nn.MaxPool2d(2),
nn.Conv2d(feat * 2, feat * 4, 3, 1, 1), nn.BatchNorm2d(feat * 4), nn.ReLU(inplace=True),
nn.MaxPool2d((2, 1)),
nn.Conv2d(feat * 4, feat * 4, 3, 1, 1), nn.BatchNorm2d(feat * 4), nn.ReLU(inplace=True),
nn.MaxPool2d((2, 1)),
)
self.rnn = nn.LSTM(feat * 4, hidden, bidirectional=True, batch_first=True)
self.head = nn.Linear(hidden * 2, vocab_size)
def forward(self, x):
# x: (N, 1, 32, 80) — a five-character line at 16 px per character
f = self.cnn(x) # (N, 128, 2, 20)
f = f.mean(dim=2).transpose(1, 2) # (N, 20, 128) ← width is now time
h, _ = self.rnn(f) # (N, 20, 256)
return F.log_softmax(self.head(h).transpose(0, 1), dim=-1) # (T=20, N, 40)
model = TinyCRNN()
print(sum(p.numel() for p in model.parameters())) # 514,408Then data. Real OCR datasets add fonts, noise, rotation, blur and colour — but the pipeline shape is identical to five lines of synthetic text, which is what makes this a laptop exercise:
import numpy as np
import torch
VOCAB = ["_"] + list("0123456789abcdefghijklmnopqrstuvwxyz") # 37 symbols
def synthetic_line(text, height=32, char_width=16):
W = char_width * max(1, len(text))
img = np.ones((height, W), dtype=np.float32) # white page
for i, c in enumerate(text):
x = i * char_width
img[6:height - 6, x + 2:x + char_width - 2] = 0.0 # black glyph block
return img
def build_batch(strings, max_len=None):
H = 32
max_len = max_len or max(len(s) for s in strings)
W = 16 * max_len
imgs = np.ones((len(strings), 1, H, W), dtype=np.float32)
targets, target_lengths = [], []
for i, s in enumerate(strings):
line = synthetic_line(s)
imgs[i, 0, :, :line.shape[1]] = line
ids = [VOCAB.index(c) for c in s]
targets.extend(ids)
target_lengths.append(len(ids))
return (torch.from_numpy(imgs), torch.tensor(targets),
torch.tensor(target_lengths))
model = TinyCRNN(vocab_size=len(VOCAB))
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
for step in range(200):
strings = [f"abc{step % 10}"] * 4 + [f"xy{(step + 1) % 10}{(step + 2) % 10}"] * 4
imgs, targets, target_lens = build_batch(strings, max_len=5)
log_probs = model(imgs) # (W', 8, vocab)
input_lens = torch.full((8,), log_probs.size(0), dtype=torch.long)
loss = ctc_loss(log_probs, targets, input_lens, target_lens)
opt.zero_grad(); loss.backward(); opt.step()
if step % 40 == 0:
print(f"step {step:3d} loss {loss.item():.3f}")Two numbers complete the picture. First, T ≥ S: the model above gives 20 time steps for 5 characters, so every target is representable; a 40-character line in a 320 px crop (8 px per character, T = 80) is fine too, but a 200-character line in the same crop is not — and with zero_infinity=True that sample contributes a loss of 0 instead of an error, which is why long-line oversampling and crop length checks matter more in production than they look.
Second, the latency budget per page. Recognition is per line, and lines batch beautifully. Take a 50-line page: one detection pass at ~30 ms, 50 line crops through the recogniser at ~2 ms each when batched — ~100 ms — and a layout pass at ~20 ms. That is 150 ms per page, or about 6–7 pages per second on one modest GPU. The end-to-end alternative does not have a per-line stage to batch: a VLM generating 3,000 characters of output is decoding ~1,200 tokens one at a time, which is seconds, not milliseconds. This is the budget line that decides most production architectures, and it is why the two approaches coexist rather than compete.
Production OCR is a purchasing decision as much as a modelling one: language coverage, latency, accuracy and cost move together, and the right answer changes with the rung of the ladder you are standing on. The rule is to buy the cheapest rung that answers the question.
Four questions, in this order, decide almost every project. Run them before you look at a leaderboard:
Mapped onto the ladder: rungs 1–2 (text, multilingual text) are PaddleOCR or EasyOCR territory, with Tesseract for CPU-only archive work; rung 3 (tables) adds a layout and table-structure model, or a VLM when tables are irregular; rung 4 (fields) is a fine-tuned Donut or LayoutLMv3; rung 5 (questions across documents) is VLM territory, ideally with OCR text and retrieval behind it so every number can be traced back to a crop.
PICK = """
You are choosing an OCR stack. Given: document_type, language_scripts,
volume_per_day, needs (text | table | fields | answers), label_budget.
Rules, in order:
1. needs == "text" and printed -> PaddleOCR (fastest open pipeline)
EasyOCR if you need a torch-native drop-in
Tesseract if CPU-only or 100+ languages
2. handwritten lines, boxes exist -> TrOCR
3. needs in ("table", "fields") -> fine-tuned Donut if label_budget >= 100
LayoutLMv3 on plain scans
4. needs == "answers" or the schema
changes per vendor -> VLM (Qwen-VL-OCR class) + retrieval
5. always: report the metric for the
rung you are on (CER / cell F1 / field F1 / answer accuracy),
not the metric the model was marketed with.
"""Five tasks, three layers: OCR, then layout, then understanding. Pick a task and see what actually solves it, what to measure, what it costs — and the one way it usually fails.
OCR gives you the cell text as a stream in reading order; the structure is a second problem. A 6 × 20 table is 120 cells, and getting 95% of them right still leaves 6 cells wrong — in the row that holds the number your report is about.
Failure mode: Merged cells and multi-line headers break the row grid; a model trained on plain tables will silently merge two columns and hand you a plausible number.
The ladder is the lesson’s rule of thumb: pick the lowest rung that answers your question. Fields on a repeating form (rung 4) mostly need fine-tuned Donut, not a VLM; a 10-million-page digitisation (rung 1) mostly needs PaddleOCR, not a VLM.
One closing observation, because it is the lesson’s motto in a different costume: the three-stage pipeline is not a historical artifact. Donut deletes the detector and the layout module but still has an encoder that sees the page and a decoder that writes an answer in order. A VLM merges all three stages and adds reasoning, but it is still detecting, recognising and ordering internally — just without exposing the seams. Understand the stages and you can read any of these systems; skip them and every model is a black box with a benchmark attached.
The blank-token question and the CTC-versus-cross-entropy question are the two you will be asked for the rest of the phase. The greedy question and the Donut question separate mechanism from marketing; the invoice_total question is the one that shows up in job interviews; and the layout question is the one that decides whether your extracted numbers are trustworthy.
0 / 6 answered · 0 correct
01Why does OCR use CTC loss instead of plain cross-entropy?
02In CTC decoding, what does the blank token actually do?
03Why is greedy CTC decoding sometimes better than beam search on simple datasets?
04Donut skips explicit text detection and character recognition. How?
05For extracting invoice_total from receipts, which approach is typically best in 2026?
06What does layout parsing add on top of OCR text?
Click a card to swap the lazy description for what it actually means.
Three problems with exact numbers — train the tiny CRNN on digit strings and report CER, swap greedy decoding for a width-5 beam search and explain the delta, then run PaddleOCR over 20 receipts and compute field-level F1. Try first; a worked answer is one click away.
Use VOCAB = blank + 10 digits (11 classes); uniform loss is −ln 11 = 2.398, so watch the first few steps land near 2.4 and fall. A 5-digit line at 16 px/char is 80 px wide, and the network downsamples width by 4 → T = 20 time steps for S = 5 characters, a 4× margin, so every target is representable. Generate 2,000 training strings and 200 held-out ones; keep the batch at 8–32 and 500 steps is plenty on trivial synthetic glyphs. Report CER as total edit distance over total reference characters: 200 strings × 5 digits = 1,000 characters. If 30 characters are wrong on the held-out set, CER = 30/1,000 = 3%. The important habit is the failure analysis: in a 5-digit string one dropped digit costs 1/5 = 20% of that string's characters, so a 3% aggregate CER usually means a handful of complete failures, not uniform noise — look at the worst ten examples first.
Implement prefixes rather than paths: keep a set of candidate output prefixes with a probability, extend each by every symbol at the next time step, and merge two extensions that differ only by a trailing blank into the same prefix. Score each prefix by the log-probability of its best path; prune to the top 5. The delta on clean synthetic data is near zero — greedy is already within ~1% CER of beam, and often identical, because the model's per-step argmax never contradicts the best prefix. Beam wins where a step is ambiguous: when two symbols sit within a few hundredths of each other, the argmax can be locally right and globally wrong, and keeping the runner-up recovers the word. Cost: beam width k is k× the decode work, but decoding is a rounding error next to the CNN+BiLSTM forward pass — the real cost appears when you decode millions of lines, or when you use a language model as a beam scorer, which is where the 5–10× figures come from. Report the delta per character, not per word, so it is comparable with the published CER numbers.
Three stages, and the error will not be where you expect. (1) Recognise: run PaddleOCR with the right language model, keep the boxes and scores. (2) Order and pair: sort lines into reading order, then parse each line as a price at the end of the string — beware the glue case where OCR emits "Consulting hours12" with no space, so anchor on the currency/decimals regex rather than on whitespace. (3) Score: take 20 receipts × 12 items = 240 expected pairs. If the system predicts 230 pairs with 224 matching the ground truth (normalise prices as numbers: 1,240.00 == 1240), precision = 224/230 = 97.4%, recall = 224/240 = 93.3%, F1 = 2 · 0.974 · 0.933 / (0.974 + 0.933) = 95.3%. The losses split cleanly: recall misses are lines the detector dropped (low-score table rows, as in the lesson's detection board) or items split across two boxes; precision misses are rows where a header line got parsed as an item, or the tax line matched the price pattern. A classical pipeline's F1 here is usually 90–97% and the ceiling is the detector's line recall — the recogniser is rarely the bottleneck. That is exactly why field extraction moved to fine-tuned Donut, which optimises the field directly instead of hoping the line-level stages combine into it.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 04, Lesson 19) and the Math Foundations Notebook reference build. The five labs (the canvas CTC decoder, the canvas detection board with its IoU inspector, the canvas stack comparator, the document-task ladder and the output formatter) are original to this page, as are the ladder arithmetic (a 3,000-character page, 2% CER = 60 wrong characters, the ~500-word denominator; 1,000 invoices × 4 fields = 4,000 values at 98% F1 = 80 wrong), the error-accumulation product (0.99 × 0.98 × 0.97 = 0.9411), the IoU landmark table recomputed by the labs' own iou(), the DB/CRAFT probability-map-to-quads post-process, the alignment-count formula C(T+S, 2S) with the 462-alignment and 132-cell forward–backward examples, the collapse table with the wrong-order counterexample, the T ≥ S rule and the zero_infinity=True trap, the TinyCRNN parameter count (514,408, of which the BiLSTM is 263,168 = 51%) and its 32×80 → T=20 shape trace, the −ln(40) = 3.689 chance-level loss, the per-page latency budget (30 + 50 × 2 + 20 = 150 ms) and the cost arithmetic ($0.17 and $2.78 per 1,000 pages at $2/h; ~$1.20 per 1,000 pages for a VLM at an assumed $1/M output tokens), the 6 × 20 = 120-cell table arithmetic, and the D-R-O and O < C < R memory hooks. Every number shown is computed live by the labs or verified by hand in the prose.