A vision encoder turns the image into patch tokens. A 21M-parameter projector rewrites each token into the LLM’s embedding space. The language model does the rest — and that ViT-MLP-LLM pattern is every production VLM in 2026, from an 8B model on one GPU to 241B mixture-of-experts systems that operate desktops. Seven chapters of shapes, training stages, dated model claims, DeepStack and the CMER number that keeps hallucination honest.
The encoder turns a 336×336 page into 576 patch tokens plus a class slot; the 2-layer MLP projector rewrites each token from width 1,152 into the LLM's 4,096; the decoder LLM reads image and text tokens as one sequence and writes the answer. The bridge is tiny — 21,504,000 parameters, about a quarter of a percent of the stack — and it is the part most teams actually train.
(3,336,336) → (576,1152) → (576,4096) → text336 ÷ 14 = 24, so the patch grid is 24 × 24 = 576 patches, plus the class slot most encoders carry — 577 positions; LLaVA-1.5 projects the 576 patch tokens themselves. Either way an image costs about a page of text: 577 ≈ 444 words ≈ 0.89 pages, and four screenshots are 2,308 tokens, 1.8% of a 128K window. Resolution grows the bill with the square, and video is out of reach entirely: one frame per second for an hour is 2,077,200 image tokens.
576 patches + 1 class slot · page ≈ 650 tokens · 1 fps × 1 h = 2,077,200Stage 1 freezes the encoder and the LLM and trains only the projector on image-caption pairs — 20,979,712 parameters moving, 320 MiB of optimizer state instead of 113 GB for the full stack. Stage 2 unfreezes everything on 500M+ interleaved pairs. Stage 3 instruction-tunes on (image, question, answer) triples; most production LoRA runs target stage 3. In production, CMER monitors what escapes: confident answers that the image does not support.
align (595K captions) → pretrain (500M+) → instruct (158K triples)A VLM is three boxes and two shape changes: the encoder changes the kind of thing (pixels into tokens), the projector changes a number (each token’s width), and the LLM changes the answer — with alignment training making the middle box a translator instead of a random map.
By the end you will be able to derive the image-token count of any VLM (336 ÷ 14 = 24 → 576 + 1 = 577), compute a projector’s parameters by hand (1024→4096→4096 = 20,979,712), say what each of the three training stages freezes and why, compare Qwen3-VL / InternVL3.5 / LLaVA-Next / GLM-4.6V on parameters, context and licence with dated claims, explain what DeepStack buys without adding tokens, and compute Cross-Modal Error Rate on a set of answers and act on it.
CLIP (Lesson 18) can score how well a caption fits an image, but it cannot answer “how many red cars are in this shot?” — it does not generate text. A vision-language model does, by bolting a language model onto a vision encoder through one small adapter.
Three components, and each one has exactly one job:
That is the whole architecture. In 2026 the pattern runs from an 8B dense model you can fine-tune on one GPU to 241B mixture-of-experts systems, and open models rival or beat GPT-5-class and Gemini-2.5-Pro-class hosted systems on multimodal benchmarks — MMMU, MMBench, DocVQA, ChartQA, MathVista, OSWorld. The differences between them are which ViT, which projector, which LLM, which training data and which alignment recipe. Once you see the three boxes, swapping any one of them is mechanical.
The merge is the part beginners miss. The text prompt contains a special placeholder token — <image> — once per patch. The runtime projects the vision tokens, then scatters them into the placeholder positions of the text embedding matrix, keeping the order. The LLM never knows an image was involved: it reads one sequence of 4096-wide vectors, some of which happen to have been born as pixels.
class MinimalVLM(nn.Module):
def __init__(self, vit, projector, llm, image_token_id):
super().__init__()
self.vit = vit
self.projector = projector
self.llm = llm
self.image_token_id = image_token_id # <image> in the text prompt
def forward(self, image, input_ids, attention_mask):
# 1. the image becomes tokens: (B, 576, 1152) at 336², patch 14
vision_tokens = self.vit(image)
# 2. the projector rewrites their width: (B, 576, 4096)
vision_embeds = self.projector(vision_tokens)
# 3. the text prompt becomes embeddings: (B, M, 4096)
text_embeds = self.llm.get_input_embeddings()(input_ids)
# 4. put image vectors where the <image> placeholders are
merged = self._merge(text_embeds, vision_embeds, input_ids)
# 5. the LLM reads pixels and words as one sequence
return self.llm(inputs_embeds=merged, attention_mask=attention_mask)
def _merge(self, text_embeds, vision_embeds, input_ids):
out = text_embeds.clone()
expected = vision_embeds.size(1)
for b in range(input_ids.size(0)):
positions = (input_ids[b] == self.image_token_id).nonzero(as_tuple=True)[0]
if len(positions) != expected:
raise ValueError(
f"batch item {b} has {len(positions)} image tokens, "
f"the encoder produced {expected} patches.")
out[b, positions] = vision_embeds[b]
return outWalk one page through the pattern. The vision encoder changes the image into tokens; the projector changes each token’s width; the LLM reads the merged sequence. Nothing else changes — that is the whole trick.
The projector is the only stage that changes a number the LLM cares about: token count is fixed by the encoder, width is fixed by the LLM, and the projector is the adapter between them.
The encoder half is the ViT from Lesson 14, usually pretrained against text (CLIP, SigLIP) or with self-supervision (DINOv3). Its output is not a caption and not a class label — it is a grid of patch tokens, and every one of them will occupy a seat in the LLM’s context window.
Start from the arithmetic you already know. A 336×336 image at patch size 14 divides into 336 ÷ 14 = 24 positions per side, so the patch grid is 24 × 24 = 576 patches. Most ViT designs prepend a class slot on top, which makes the encoder’s output 577 positions — and LLaVA-1.5 chooses to project the 576 patch tokens themselves, dropping the slot. Either number you read in a paper is telling the truth; the order of magnitude is always 24 × 24. Bump the input to 448² at the same patch size and it is 32 × 32 = 1,024 patches (+1 slot) = 1,025 positions — resolution grows the token count with the square, and attention cost with the fourth power.
Every open VLM picks a corner of that table. LLaVA-1.5 feeds the 576 patch tokens from CLIP ViT-L/14 at 336²; InternVL’s InternViT produces 1,024 visual tokens per 448² tile and then compresses them 4× to 256 with a pixel-shuffle module before the LLM sees them, trading spatial detail for context; Qwen3-VL keeps its custom ViT and pays for more tokens. The encoder is not a detail — it decides how much of the image survives to the tokens, and no later component can invent detail it threw away.
Compare image tokens to text tokens, because they share one budget. English runs about 1.3 tokens per word, and a typed page is about 500 words — roughly 650 tokens. One 336² image is 577 tokens, so an image costs about a page of text. Four screenshots are 2,308 tokens; a 128K window holds them in 1.8% of its space, which is why document pipelines are rarely context-starved. Video is the opposite: one frame per second for an hour is 3,600 × 577 = 2,077,200 tokens — no 2026 context window holds it, so every video VLM samples frames and timestamps them.
Shape: 336 ÷ 14 = 24; 24 × 24 = 576 patches; plus the encoder’s class slot = 577. In bytes: 336 × 336 × 3 = 338,688 pixel values become 577 × 1,152 = 664,704 floats at the projector input — the encoder is not compressing for the LLM, it is re-shaping for it.
Money: 577 tokens ≈ 577 ÷ 1.3 = 443.8 words ≈ 0.89 pages. Four images ≈ 2,308 tokens, which is 2,308 ÷ 128,000 = 1.8% of a 128K window and 2,308 ÷ 32,000 = 7.2% of a 32K window.
Patience: every one of those tokens attends to every other token in the LLM. 577 extra seats do not just cost 577 units of prefill; they add 577 × (M + 577) attention pairs, which is why the first thing a serving stack does with a big image is crop it.
Choose the page, the patch size, how many images and how much text. Bar one shows your request; bar two shows it inside the context window, to scale. Image tokens are not free — an image costs about a page of text, and video costs every window you have ever heard of.
Patch size is the cheapest dial: 336² at patch 16 costs 442 tokens instead of 577 at patch 14 — 23% fewer tokens for less spatial detail. Resolution is the expensive dial: it grows with the square.
A 448×448 scanned page goes through an encoder with patch size 16. How many tokens reach the projector, including the class slot?
The projector is the smallest of the three components and the one most teams actually train. Its job is narrow: take each vision token and rewrite it so the language model reads it as just another word vector.
The shape walk makes the job obvious. The encoder hands over (576, 1152) — 576 patch tokens, each 1,152 numbers wide, because that is what SigLIP so400m outputs. The LLM was born expecting (M, 4096), because that is the width of its embedding table. The projector is the function between them: (576, 1152) → (576, 4096). The token count does not change, the order does not change, no patch is mixed with any other. Every token is widened independently — a pointwise map, not a reasoning step.
The standard implementation is almost embarrassing: two linear layers with a GELU in between. LLaVA-1.5 calls it the MLP connector — Linear(1024→4096) + GELU + Linear(4096→4096) — and that is 20,979,712 parameters, about three tenths of one percent (0.297%) of the full encoder + projector + LLM stack. The alternative lineage is the Q-former: BLIP-2’s ~188M-parameter transformer with 32 learned query tokens that cross-attend to the image features and emit a fixed 32-token summary. It caps the image budget at any resolution, but it throws spatial detail away and needs its own pre-training. For instruction-tuned assistants the plain MLP won.
Layer one, Linear(1024 → 4096): 1024 × 4096 weights = 4,194,304, plus 4,096 biases = 4,198,400.
Layer two, Linear(4096 → 4096): 4096 × 4096 = 16,777,216 weights, plus 4,096 biases = 16,781,312.
Total: 4,198,400 + 16,781,312 = 20,979,712. Compare it to its neighbours: CLIP ViT-L/14 is 303,179,776, the Vicuna-7B language model is 6,738,415,616, so the stack is 7,062,575,104 and the projector is 0.297% of it. LLaVA-NeXT’s SigLIP + Llama-3-8B pairing is 1152 → 4096 → 4096 = 21,504,000 — effectively the same bridge at the same price. This is why projector-stage training is the cheap lever, and why a bigger projector is almost never the fix when a VLM is wrong.
class Projector(nn.Module):
"""d_vit → hidden → d_llm, applied to every patch independently.
No attention, no mixing across patches — just a width change."""
def __init__(self, vit_dim=1024, llm_dim=4096, hidden=4096):
super().__init__()
self.net = nn.Sequential(
nn.Linear(vit_dim, hidden),
nn.GELU(),
nn.Linear(hidden, llm_dim),
)
def forward(self, x):
# x: (N_patches, d_vit) -> (N_patches, d_llm)
return self.net(x)
# LLaVA-1.5's connector, exactly:
p = Projector(1024, 4096, 4096)
print(sum(t.numel() for t in p.parameters())) # 20,979,712
# shape trace for one 336² image
vision = torch.randn(576, 1024) # CLIP ViT-L/14 features
print(p(vision).shape) # torch.Size([576, 4096])Six patches on the left, the six words they should mean on the right. The 2×2 matrix in between is trained live by gradient descent on those pairs — the same objective a real projector runs, shrunk until you can see it. Teaching model: production projectors are 4096-wide MLPs.
The lesson inside the toy: the bridge never changes what a token is — only where it points. Training is what turns a random 2×2 (or a random 21M-parameter MLP) into a map whose output lands near the words the LLM already understands.
A projector maps a (576, 1152) tensor to (576, 4096). What changed, and what did not?
Nobody trains a VLM end-to-end from random weights any more than they train a transformer from scratch. The recipe is three stages, and the first one is almost free: freeze everything, train the 21M-parameter projector on image-caption pairs.
Stage 1 — alignment. The vision encoder and the LLM are frozen. Only the projector trains, on hundreds of thousands of image-caption pairs, and its loss is simply “say the caption that goes with this image”. The job it is learning is the dialect change: where a patch that means “golden retriever head” has to land so that the LLM’s word embedding for dog lights up. The LLaVA paper used 595K caption pairs here. Because the frozen stack does the thinking and only 20,979,712 parameters move, this stage fits on a single GPU and is the stage most teams re-run on their own images.
Stage 2 — pre-training. Unfreeze everything and train on large-scale interleaved image-text data — the source’s scale is 500M+ pairs. This is where visual knowledge is built: object names, chart conventions, document layouts, the visual half of world knowledge. It is expensive, it is why frontier VLM labs exist, and it is the stage you will never run on a laptop.
Stage 3 — instruction tuning. Fine-tune on curated (image, question, answer) triples. The LLaVA paper used 158K of them; the model learns not what an image is but how to behave about one: follow the format, answer the actual question, say “I can’t tell from this image” when that is true. This is the stage that turns a vision-aware language model into a usable assistant. Most production LoRA fine-tunes target stage 3 with 5,000–50,000 labelled examples, rank 16–64 adapters, $100–$5,000 of compute and 2–10 hours of training.
The three-stage recipe, drawn as who moves. Stage 1 is the alignment stage from the quiz: frozen encoder, frozen LLM, one small bridge learning to translate. Stages 2 and 3 unfreeze the stack — and stage 3 is what every production fine-tune imitates with LoRA.
Full fine-tuning needs, per parameter: a gradient (4 bytes), the two AdamW moment buffers (8 bytes) and usually an fp32 master copy (4 bytes) on top of the weights themselves — about 16 extra bytes per parameter. For the 7,062,575,104-parameter LLaVA-1.5 stack that is 113.0 GB of optimizer and gradient state, before activations.
The alignment stage moves 20,979,712 parameters, so its state is 20,979,712 × 16 = 335,675,392 bytes = 320 MiB. Three orders of magnitude less, and no activation memory for the frozen transformer blocks because they run without gradients. That is the real reason the alignment stage is cheap: not the FLOPs, the memory.
LoRA at stage 3 lands in between: adapters of rank 16–64 on the attention projections add millions of parameters, not billions, so a 70B VLM becomes tunable on a single H100 while the base weights stay in whatever precision you loaded them in.
# Stage 1 — alignment: only the bridge learns.
for p in vit.parameters(): p.requires_grad = False
for p in llm.parameters(): p.requires_grad = False
for p in projector.parameters(): p.requires_grad = True
# data: (image, caption) pairs; loss: next-token on the caption
# Stage 2 — pre-training: everything learns on a large corpus.
for p in model.parameters(): p.requires_grad = True
# data: 500M+ interleaved image-text samples; this is the expensive stage
# Stage 3 — instruction tuning: everything (or LoRA + projector) on triples.
# data: (image, question, answer); loss: next-token on the answer only
# production version — 5k-50k examples, 2-10 hours, one accelerator:
for name, p in model.named_parameters():
p.requires_grad = "lora" in name or "projector" in nameQwen3-VL, InternVL3.5, LLaVA-NeXT and GLM-4.6V look like four different models. Read their columns and they are four answers to the same three questions: which vision encoder, which projector, which LLM — plus how much data and which alignment recipe.
The source’s early-2026 snapshot, which the comparator lab dates line by line:
Two rows deserve their own paragraph. The Qwen3-VL family is where the pattern is most visible: the 235B MoE activates 22B parameters per token, uses a custom ViT with DeepStack, and its report claims state-of-the-art GUI grounding (its 32B posts 41 on OSWorld). That is what a visual agent is — the model receives a desktop or mobile screenshot, understands the UI, and emits click / type / scroll actions through tool calls. Combined with tools it closes the loop on common desktop tasks, which is the machinery under most 2026 “AI PC” demos and a direct route into automated QA, RPA and accessibility.
Video adds a second axis the pattern did not have to think about: when did this frame happen? Qwen3-VL evolved from temporal rotary embeddings (T-RoPE) to explicit text-based time alignment — timestamp tokens interleaved with the frames, so the model reads <timestamp 00:32> next to a frame and can reason about temporal order like any other text. And the honest limitation: current VLMs score about 50–60% on spatial-reasoning benchmarks (above/below, left/right, counting, distance) — below human. If your task is purely spatial, validate heavily or use a dedicated keypoint, depth or detection model instead.
from transformers import AutoProcessor, AutoModelForVision2Seq
import torch
from PIL import Image
model_id = "Qwen/Qwen3-VL-8B-Instruct"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForVision2Seq.from_pretrained(
model_id, torch_dtype=torch.bfloat16, device_map="auto")
messages = [{
"role": "user",
"content": [
{"type": "image", "image": Image.open("chart.png")},
{"type": "text", "text": "What does this chart show?"},
],
}]
inputs = processor.apply_chat_template(
messages, add_generation_prompt=True, tokenize=True,
return_dict=True, return_tensors="pt").to("cuda")
generated = model.generate(**inputs, max_new_tokens=256)
answer = processor.decode(
generated[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(answer)The 2026 open-VLM landscape as the lesson’s source describes it, row by row, with the official model card’s reading next to it. Parameters, context length and licence decide deployment before any benchmark does.
| model | params | context (source) | context (card, Sep 2026) | licence |
|---|---|---|---|---|
| 235B total · 22B active | 256K | native 256K, expandable to 1M | Apache-2.0 | |
| 30B total · 3B active | 256K | native 256K, expandable to 1M | Apache-2.0 | |
| 8B | 128K | native 256K, expandable to 1M | Apache-2.0 |
vision · custom ViT + DeepStack | language · Qwen3
strength: General SOTA; state-of-the-art GUI-grounding claims; OSWorld 41 reported for the 32B
watch out: MoE serving needs real infrastructure — 235B of weights must be resident even though only 22B activate per token
Qwen3-VL-235B-A22B model card ↗MoE + DeepStack; the 2026 general-purpose default and GUI-agent winner
your jobModel claims are perishable. The source table and the current cards disagree on context for three of these rows — every number above is dated, and the model card wins when they disagree. Licences are for the weights as published; check the base model’s terms too.
Vanilla projection reads the last ViT layer, where the features are most semantic and least spatial. DeepStack samples several depths and stacks them along the channel axis, so the LLM gets the meaning and the coordinates in the same token.
A ViT is not a function, it is a stack of them. Early layers carry fine-grained spatial and textural detail — where the bar is, what colour it is, whether there is a thin border. Late layers carry semantics — this is a revenue chart. Project only the last layer and you hand the LLM a summary; the coordinates were overwritten on the way up. DeepStack keeps both: sample features from several depths, concatenate them along the channel axis, and let the projector map the wider vector into the LLM’s space.
The arithmetic stays friendly. Three levels of a 1,152-wide encoder concatenate to 3,456 numbers per patch, and the projector’s first layer reads those instead of 1,152: Linear(3456→4096) + Linear(4096→4096) = 30,941,184 parameters instead of 21,504,000 — a 1.44× cost on a component that is still 0.4% of the stack. Four levels = 35,659,776. Crucially, the token count never moves: 577 tokens before DeepStack, 577 after. What changed is how much each token knows about both the meaning and the place.
DeepStack, drawn at 1,152-wide features. Shallow taps (left) carry where, deep taps (right) carry what; concatenation gives the projector both, and the LLM’s sequence length stays 577.
# sampled_features: [f_layer4, f_layer8, f_layer12]
# each: (B, 577, 1152) for one 336² image
fused = torch.cat(sampled_features, dim=-1) # (B, 577, 3456)
projector = nn.Sequential(
nn.Linear(1152 * 3, 4096), # fc1 grows with the number of taps
nn.GELU(),
nn.Linear(4096, 4096), # fc2 is unchanged
)
image_tokens = projector(fused) # (B, 577, 4096)Your model adds DeepStack with three ViT levels. How long is the image-token sequence, and what changes?
About 12% of image-text pairs in a crawled dataset contain descriptions that are not fully grounded in the image. A VLM trained on them learns to fabricate fluently — invented objects, misread numbers, relationships that were never there. In production this is the dominant failure mode, and the source’s answer is to measure it.
The failure has a shape. A grounded answer and a fabricated one sound identical, but they differ on two axes you can compute: how confident the model was in its text (the mean per-token probability), and how similar that text is to the image (a cosine in a CLIP-family embedding space). Cross-Modal Error Rate is the fraction of outputs sitting in the corner where confidence is high and similarity is low:
The point is not the exact number. It is that the number is monitorable: per endpoint, per prompt type, per customer. A rising CMER says the model has drifted out of distribution or met a prompt category it cannot handle — on that input set, today. The source’s deployment made it a first-class KPI and cut hallucination about 35% without changing the model, because the action is routing, not retraining: high-CMER outputs go to human review.
Mitigations stack, and each catches a different failure shape. Grounding instructions (“answer only from what is visible”) remove the unanswered-question failures. A refusal policy lets the model say “not visible in this image”. A logprob re-check catches the overconfident few before they leave the service. Human review catches everything that is flagged — and costs reviewer minutes, which is exactly what keeps it from being free. What does not work is tightening thresholds until the dashboard looks good: you have not grounded the model, you have hidden it.
import torch.nn.functional as F
def cross_modal_error_rate(image_emb, text_emb, text_confidence,
sim_threshold=0.25, conf_threshold=0.8):
"""Fraction of outputs that are confident AND ungrounded.
image_emb, text_emb: embeddings of the image and of the generated text
text_confidence: mean per-token probability of the answer, in [0, 1]"""
image_emb = F.normalize(image_emb, dim=-1)
text_emb = F.normalize(text_emb, dim=-1)
sim = (image_emb * text_emb).sum(dim=-1) # cosine similarity
high_conf_low_sim = (text_confidence > conf_threshold) & (sim < sim_threshold)
return high_conf_low_sim.float().mean().item()
# with 8 outputs, 4 of them confident and ungrounded:
# CMER = 0.500 — and a per-1,000 figure of 500 review candidatesEight generated answers, each with a mean token probability (text confidence) and a CLIP-family image-text similarity. CMER flags the rows that are confident and ungrounded. Then switch the four mitigations on — the lesson’s reading of the source’s playbook — and watch what each one actually fixes.
| # | answer | conf | sim | board verdict |
|---|---|---|---|---|
| 1 | What does the chart show? Revenue rises from 2021 to 2024. | 0.95 | 0.71 | grounded |
| 2 | How many bars are in the chart? There are five bars. | 0.90 | 0.64 | grounded |
| 3 | What colour is the tallest bar? The tallest bar is green. | 0.88 | 0.14 | FLAGGED |
| 4 | Is there a title on the page? Yes — “Quarterly Revenue”. | 0.85 | 0.68 | grounded |
| 5 | What is the value of the last bar? The last bar reads 42. | 0.92 | 0.09 | FLAGGED |
| 6 | Is the chart a pie chart? No, it is a bar chart. | 0.90 | 0.58 | grounded |
| 7 | What is the manager's name? The manager is J. Alvarez. | 0.87 | 0.19 | FLAGGED |
| 8 | How many pages are in the document? The document has 12 pages. | 0.91 | 0.23 | FLAGGED |
Similarity is the image-text cosine from a CLIP-family checker; confidence is the mean per-token probability of the answer. CMER flags only the top-right corner of that grid: confident and ungrounded. The board’s verdict column is the ground truth for this toy set — in production you never have it, which is why CMER is a monitor rather than a fix.
Two lessons from the buttons. Grounding and refusal are cheap and fix specific failure shapes; logprob checks catch the overconfident few; human review catches everything but costs minutes. Then loosen the similarity threshold far enough (0.25 → 0.10) and the dashboard looks perfect because ungrounded answers fell out of the monitor — not out of the users’ screens. CMER is a monitor, not a fix.
With thresholds conf > 0.80 and sim < 0.25, which answer does CMER flag? A: confidence 0.93, similarity 0.11. B: confidence 0.71, similarity 0.09.
The projector question and the DeepStack question are the two the pattern lives on. The CMER question is the one that shows up in production interviews; the encoder question explains why nearly every VLM starts from CLIP or SigLIP; the OSWorld question is where the field is going next.
0 / 5 answered · 0 correct
01In the ViT-MLP-LLM pattern, which component is most commonly trained while the others are frozen during the alignment stage?
02DeepStack (used in Qwen3-VL) does what?
03A production VLM shows high text confidence but the generated text describes objects that are not in the image. Which metric captures this failure?
04Why do modern VLMs use SigLIP or a custom CLIP-family vision encoder rather than a supervised ImageNet ResNet as the backbone?
05Qwen3-VL's technical report claims state-of-the-art GUI grounding. What does it actually report for OSWorld, the end-to-end desktop-control benchmark?
Click a card to swap the lazy description for what it actually means.
Four problems with exact numbers — score a first-pass CMER by hand, budget six scanned pages and a projector to the byte, fine-tune a small VLM with LoRA and report what actually changed, and swap the encoder to DINOv3 to see the dense-vs-VQA trade. Try first; a worked answer is one click away.
Take the 15 answers and tag each one: grounded (the answer is supported by the pixels) or ungrounded (it invents an object, a number or a relation). A realistic first run on uncurated photos looks like 11 grounded and 4 ungrounded → a first-pass rate of 4/15 = 26.7%. Then refine the same way the code does: instead of your own judgement, record the mean per-token probability of each answer (the API or the local model returns logprobs) and the cosine similarity between the image embedding and the answer text from a CLIP-family model. Flag answers with confidence above 0.8 and similarity below 0.25 and count them. The number will not equal your hand count — that gap is the whole point of CMER: it is a cheap automatic proxy for a judgement you cannot run on every request. Do the exercise twice on different prompt sets; “count the objects” produces most of the hallucinations, which is exactly the counting weakness the lesson warns about.
First the token arithmetic: 448 ÷ 14 = 32, so each page is a 32×32 grid = 1,024 patches, plus a class slot = 1,025 tokens (6,144 tokens across the six pages if the encoder drops the slot, as LLaVA does — six tokens do not change the design). Six pages = 6,150 image tokens; add the 1,500-token prompt → 7,650 tokens = 7,650 / 32,768 = 23.4% of the window. If the answer needs 1,000 more tokens there is still room — 31 pages' worth of image tokens fit in 32K, which is why document pipelines are bottlenecked by accuracy, not context. The projector: Linear(1152→4096) is 1152 × 4096 + 4096 = 4,722,688; Linear(4096→4096) is 4096 × 4096 + 4096 = 16,781,312; total 21,504,000 parameters, about a quarter of a percent of a CLIP-L/SigLIP + 8B stack. At patch 28 the grid is 16×16 = 256 patches + 1 = 257 tokens per page — 6 × 257 = 1,542 tokens, 4.7% of the window — because each patch now averages 28×28 pixels instead of 14×14: four times fewer tokens, four times less spatial resolution. Fine print (invoice numbers, table cells) disappears, so the honest fix for detail is a crop of the region at patch 14, not a bigger page. Frozen: during alignment the encoder and the LLM stay frozen and only the projector trains, which is exactly why this pipeline can be adapted on one GPU with a few thousand labelled pages.
Load a 3B–7B VLM (Qwen2.5-VL-3B or LLaVA-1.6-7B), attach LoRA rank 16, alpha 32, dropout 0.05 to the attention projections and the projector, and train in bf16 with lr 2e-4 cosine and 2–3 epochs over 500 examples — 1,500 example-passes, roughly 375 optimizer steps at batch 4. Hold out 100 in-domain images and 100 out-of-domain images before you start. Expected shape of the result: zero-shot in-domain accuracy is poor on your domain vocabulary (product codes, part numbers, chart conventions) and respectable out-of-domain; after fine-tuning, in-domain jumps tens of points while out-of-domain usually drifts down a little. That is the honest finding to report: 500 examples buy format, vocabulary and task shape, not general capability. Measure with exact-match or a rubric, not vibes, and always keep the zero-shot model as the baseline you ship against. The source's budget for this class of run: 5,000–50,000 examples, rank 16–64, $100–$5,000 of compute, 2–10 hours.
The swap is mechanical once the shapes line up: the projector's first layer changes from Linear(d_vit_old → hidden) to Linear(d_vit_new → hidden), everything else stays. For a 1024-wide DINOv3 feeding a 4096-wide LLM through a 2-layer MLP that is the same 20,979,712-parameter projector as LLaVA's; for a 1536-wide encoder it becomes 1536 × 4096 + 4096 + 16,781,312 = 23,076,864. Freeze the LLM and DINOv3, train the projector on image-caption pairs (the LLaVA-1.5 alignment recipe), then evaluate three things separately: caption quality, visual question answering, and dense tasks (counting, above/below, left/right, pointing). Expect the dense tasks to improve — DINOv3's self-supervised features preserve spatial structure that contrastive encoders blur — and overall VQA to get worse until the projector has seen far more pairs, because DINOv3 never learned to sit next to text. That trade is the lesson: the encoder decides what survives to the tokens, and the projector cannot invent grounding that the encoder threw away. Report both directions; do not hide the regression on the task you were not optimising.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 04, Lesson 25) and the Math Foundations Notebook reference build. The five labs — the architecture explorer, the image-token budget calculator, the toy projector playground (which trains a real 2×2 projector by gradient descent), the dated model comparator and the CMER hallucination board — are original to this page, as are the exact numbers: 336 ÷ 14 = 24 → 576 + 1 = 577 tokens, 577 ≈ 444 words ≈ 0.89 pages, the projector budgets (20,979,712 for 1024→4096→4096 and 30,941,184 for three DeepStack taps), the 320 MiB vs 113 GB training-state comparison, the DeepStack paper's benchmark deltas, and the model-card readings checked in September 2026. Every number shown is computed live by the labs or verified by hand in the prose.