A depth map is a single-channel image where each pixel is a distance from the camera. For decades that channel required stereo or LiDAR. In 2026 a self-supervised ViT encoder and a DPT-style decoder get within a few percent of ground truth from one ordinary photograph — and the same map, back-projected through four numbers of camera intrinsics, becomes a 3D point cloud.
H × W × 1: at 518 × 518 that is 268,324 values, 1.07 MB as float32 or 268 KB as uint8. Near = light is a display choice, not data. Relative models give ordering and ratios; metric models give metres; a depth of 12.0 and a depth of 1.2 can look identical after normalisation.
depth ∝ 1/disparity · ratios survive, units do notDINOv2's self-supervised features encode scene structure, texture gradients and object semantics — none of it trained on depth labels. 518 / 14 = 37 patches per side, so 1,369 tokens go into a DPT-style decoder that regresses one depth per pixel. The head is what the depth data shapes most: DA3 fine-tunes the encoder at 5e-6 and its head at 5e-5, and freezes the backbone only in the 3DGS extension.
37 × 37 = 1,369 tokens → H × W × 1 depthX = (u − cₓ)·Z/fₓ and Y = (v − c_y)·Z/f_y, with Z the depth — one multiply and one divide per pixel, and the whole cloud is in the camera's frame. A detection box becomes one 3D point at its centroid, or a 3D box when you also use its size. No calibration? Assume 60–70° FOV for visualisation, never for measurement.
X = (u − cₓ)·Z/fₓ · Y = (v − c_y)·Z/f_y · Z = ZThe picture supplies ordering and ratios, the encoder supplies priors learned from billions of images, the head turns those into one number per pixel, the intrinsics turn numbers into metres, and scale alignment is the only honest way to compare a relative prediction with a ground-truth ruler.
By the end you will be able to say which models produce relative depth and which produce metres (MiDaS, Marigold, Depth Anything vs ZoeDepth, UniDepth, Metric3D); explain how a pretrained DINOv2 encoder plus a dense depth head generalises without depth labels — and where the backbone really gets frozen (the 3DGS extension); name the monocular cues that make depth possible and the four things it cannot recover; read the intrinsics formula and back-project a pixel by hand ((400, 340) at Z = 3 m is (0.88, 0.51, 3.00) m on this lesson’s camera); compute AbsRel and δ < 1.25 with and without scale-and-shift alignment (0.513 → 0.0175 on the worked example); and lift a 2D detection to a 3D point with a single depth value per box.
An RGB frame tells you where every surface projects. It does not tell you how far away anything is — and that one missing axis is the difference between a photo and a scene.
A camera flattens the world. Every point in the scene lands on a pixel, and the depth — the distance along the viewing ray — is thrown away by the projection. Two different worlds produce the same image: a 1.2 m crate at 3 m and a 4.8 m object at 12 m are pixel-for-pixel identical if everything else scales with them. A depth map is the axis put back: an image with the same width and height as the frame, but a single channel instead of three, where the value at each pixel is the distance from the camera to whatever surface that pixel sees.
For most of computer vision’s history that channel came from hardware. Stereo rigs triangulate a point from two views; LiDAR times a laser’s round trip; structured light measures a projected pattern’s deformation. Each is expensive, fragile or range-limited — and each is a second sensor to calibrate, synchronise and carry. Monocular depth estimation is the bet that the depth is already implied by the picture, and that a big enough model trained on enough images can read it out.
The stereo column is the whole case for monocular models. A depth estimate from triangulation is only as good as the disparity measurement divided by the baseline: at half a pixel of disparity error, the depth error grows as Z² — 1.7% at one metre, 17.4% at ten, useless at a hundred. Monocular models do not triangulate anything: they learn a mapping from appearance to distance, so their error does not blow up with range in the same way. That is why a phone can now fake a portrait-mode blur with one lens and why a robot can estimate how far the wall is from a single cheap camera.
Two names to keep straight from the start. A relative depth map gives ordered values — pixel A is closer than pixel B, and the ratios are meaningful — but no unit: the number 0.42 says nothing until you know what it is 0.42 of. A metric depth map gives metres. The distinction is not academic: it decides which model you pick, what loss it trains with, and whether you are allowed to compare the output to a tape measure.
Move the crate. The colour image changes as perspective dictates; the depth map changes with it — one number per pixel, and the crate’s depth moves along the hyperbola 1/Z.
The scene is a teaching cartoon with exact geometry, drawn with the lesson’s pinhole camera (fx = 480 px, 518-wide crop). The inverse law is not a coincidence: stereo triangulates 1/Z, which is why depth estimates get coarse with distance and disparity errors multiply by Z².
A depth model outputs 0.8 for a chair and 0.4 for a wall behind it. What can you say without any extra information?
The rest of the lesson follows the depth map through its whole life. Which models produce which kind of depth and why (02); why a single image contains any depth signal at all (03); the things no single image can ever contain (04); the four numbers that lift a pixel into a 3D point (05); the two metrics and the alignment step that make relative predictions comparable (06); and the pipeline that turns a detector’s box plus a depth map into a 3D point (07).
Every production depth model answers one question: metres, or ordering? The architecture underneath is nearly always the same — a strong pretrained encoder and a small depth head — and the difference lives in the data and the loss.
Relative depth is what a model learns when its supervision is photometric or scale-invariant: pixels get ordered values and the loss does not care whether the whole map is stretched by a constant. MiDaS (paper 2019–2020, v3.1 in 2023) made this the default by mixing many datasets and evaluating after alignment. Marigold (2023) reframed the same target as conditional diffusion and produced the sharpest boundaries of the era. Depth Anything (V2 2024; V3 released 14 Nov 2025, ICLR 2026) scales the training data — 1.5 M labelled and 62 M unlabelled images — and became the model to reach for when the input is an arbitrary photo.
Metric depth is what a model learns when its targets are metres: ZoeDepth (2023) trains relative-depth features and then regresses metric bins, which is why it carries separate indoor and outdoor heads — the scale statistics of a bedroom and a highway are different distributions. UniDepth (2024) goes further and estimates the camera intrinsics along with the depth. The trade for metres is exposure: a metric model is only as good as its intrinsics and its domain coverage.
The phrase “a strong pretrained encoder plus a lightweight head” is the heart of the modern recipe. The encoder’s job is to have already learnt what the visual world looks like — edges, textures, object parts, scene layout — from self-supervised pretraining on billions of images. DINOv2 did that with no labels at all: it was trained to give the same features to two augmented views of the same image. Those features turn out to encode exactly the structure that correlates with depth, so the head only has to learn the geometry: one linear-ish mapping from features to distances. In practice the encoder is not frozen — it is fine-tuned at a much lower rate than the head (Depth Anything 3: 5e-6 for the encoder against 5e-5 for the decoder), and that 10× gap is what keeps the broad pretrained features from being overwritten by the narrower depth dataset while the head learns the task. The encoder’s cross-domain generalisation — indoor, outdoor, medical, satellite — survives the fine-tune; the head is where the geometry lives.
It is worth saying plainly where the generations differ, because a quiz question has to survive contact with the actual code. Depth Anything V1 and V2 fine-tuned their DINOv2 encoders. DA3’s recipe is minimal rather than frozen: one plain transformer — a vanilla DINOv2 encoder with no architectural specialisation — plus a Dual-DPT head that predicts depth and ray maps from the same features, trained end to end. The frozen backbone shows up one level up: the feed-forward 3DGS extension initialises from the pretrained DA3 weights, freezes them, and tunes only the added GS-DPT head. What is constant is the division of labour: the encoder brings priors that generalise, the head brings the metric the task requires.
import numpy as np
from PIL import Image
from transformers import pipeline
pipe = pipeline(task="depth-estimation", model="depth-anything/Depth-Anything-V2-Small-hf")
image = Image.open("street.jpg").convert("RGB")
out = pipe(image)
depth_np = np.array(out["depth"]) # H x W, float or uint8 depending on the pipeline
scale = depth_np.max() # relative: normalise to inspect, never to measure
normalised = depth_np / (scale if scale > 0 else 1)Pick the constraint you actually face. Relative or metric is the first fork; latency and boundary quality decide the rest. Every card is a dated example — the kind of model, not a permanent ranking.
Dates matter because this field is a few years old in production terms: MiDaS set the baseline (2019–2023), ZoeDepth and Marigold split metric from quality (2023), Depth Anything became the default (2024–2026). Read the model page before trusting a ranking.
relative depth · up to scale
BEiT-L/16 in v3.1; the 2020 TPAMI paper used a ResNet — 2019–2023
speed · tens of milliseconds on a GPU; runs on a phone
detail · smooth and robust, but visibly softer at object boundaries than the 2024+ models
best for · legacy baselines, embedded and CPU-only targets, and the depth-conditioned ControlNet lineage
isl-org/MiDaS ↗relative depth · up to scale
Stable Diffusion 2 U-Net run as conditional image-to-image translation
speed · 10–50 denoising steps — roughly 10× slower than a feed-forward model
detail · the sharpest boundaries of the four; affine-invariant output, no metres
best for · offline quality work: compositing, matting, depth-conditioned generation
marigoldmonodepth.github.io ↗metric depth · metres
DPT/BEiT with metric bins and separate indoor/outdoor heads
speed · fast and feed-forward — no denoising loop
detail · outputs metres, but is sensitive to intrinsics and to the indoor/outdoor domain
best for · metres in a scene you control: measurement, AR placement, robot reach
arXiv 2302.12288 ↗relative depth · up to scale
plain DINOv2 encoder + Dual-DPT head; V2-S 24.8 M, V2-L 335.3 M, DA3-BASE 0.12 B (dated)
speed · V2-Small INT8 ≈ 30 fps at 518 × 518 on a consumer GPU
detail · the generalisation benchmark of the era; V3 accepts any number of views, with or without poses, and also ships a metric variant
best for · the default for arbitrary photos and video: no calibration, no intrinsics, any domain
depth-anything.github.io ↗DINOv2 was never trained on depth. Why do its self-supervised features still help a depth head so much?
One more distinction from the source worth keeping: Depth Anything V3 is described as predicting spatially consistent geometry from an arbitrary number of visual inputs, with or without known camera poses. In practice that means the same model architecture serves a single photo (this lesson’s main case) and a short video or photo set, where the extra views buy consistency the single-frame model cannot have. Relative depth from one image is the foundation; the multi-view case is the same idea with a stronger constraint.
Recovering depth from one view is mathematically ill-posed — and empirically possible, because real scenes are not arbitrary. Five cues correlate with distance, and a ViT trained on enough images learns the correlations.
Perspective. Parallel lines in 3D converge in 2D. A 1 m-wide lane at 5 m spans f·1/5 = 96 px on this lesson’s camera (f = 480 px), at 10 m it spans 48 px, at 20 m it spans 24 px, and every such pair meets at a vanishing point. The convergence is a geometric consequence of the projection, so the network does not have to memorise it — the flat floor plane is enough of a prior to exploit it.
Texture gradient. Surfaces carry texture at a roughly constant physical scale, so the projected scale shrinks with distance: a 1 m patch of floor spans 480 px at 1 m and 48 px at 10 m. When texture density changes smoothly across a surface, that gradient is a distance ruler — the one a blank wall removes.
Occlusion order. Nearer surfaces cover farther ones, and the T-junction where a boundary ends tells you which side is in front. It is an ordering cue rather than a distance cue, but ordering is exactly what a relative model needs, and it costs nothing to extract.
Size constancy. Familiar objects bring approximate scale: a 1.7 m person spans 255 px at 3.2 m, 85 px at 9.6 m, 26 px at 31 m. When the pixels look like a person, the prior says “person-sized”, and the projected size resolves the ambiguity between small-and-near and large-and-far. This is the cue that quietly supplies the scale monocular models otherwise lack — and the reason it works only for objects the model has seen before.
Atmospheric perspective. Outdoors, distance scatters light: contrast falls and colours shift toward the haze. A network trained on outdoor photos picks the correlation up even though it is a property of the atmosphere, not the geometry.
The last paragraph is the honest version of “the network understands 3D”. Each cue is ambiguous on its own — a large object far away projects exactly like a small object nearby — but the cues co-occur in real scenes, and training on millions of images learns the joint distribution. That is why the model can be confident and right on a street it has never seen, and confident and wrong on a mirror: the same statistics that make the guess work are the reason the failure is invisible.
It also explains the shape of the training signal. The model is not asked to produce metres it cannot know; it is asked to produce values whose ratios match the truth, and the loss is written to ignore the global scale. The result is a map that is stable and useful — and has to be aligned before any ruler touches it.
Which of these is NOT available to a model looking at a single still image?
Five cues, all static, all present in a single frame. What is missing is any guarantee: each one can be defeated by the right surface, and the next chapter is the list of defeats — scale, occlusion, reflection, sky — the places where the statistics of real scenes stop helping and a depth model is confidently wrong.
A depth model is confidently wrong exactly where the picture stops containing the information — and no amount of extra data puts the information back. Learn the four, and you can predict the failures before you see them.
1. Absolute scale. Every cue from the last chapter is invariant to a global rescaling of the world: move every surface twice as far and make every object twice as big, and the image is unchanged. A relative model returns the ordering; a metric model guesses the scale from experienced object sizes and scene statistics, which is why its errors are domain-shaped. The lesson’s scale bench shows the arithmetic: the same relative map read at ×4 reports AbsRel 2.99 and not one pixel inside δ < 1.25³, while alignment recovers 0.0145. The model was not wrong about the structure — you were reading units that were never there.
2. Occluded geometry. The back of a chair, the space behind a counter, the far side of a lamp: nothing in the frame constrains it. A network can hallucinate a plausible completion (it has seen many chairs) but there is no measurement to be had. Downstream code that assumes the depth map is a full 3D scene will collide with the parts that were never observed.
3. Reflective and textureless surfaces. A mirror shows a scene with its own depth; glass shows what is behind it; a blank wall has no texture gradient at all. The bench’s numbers: a glass surface at 2.40 m is predicted at 5.80 m because the model reads the reflected hallway (AbsRel 1.42); a textureless wall at 4.00 m drifts smoothly from 2.70 m to 5.30 m because there is no ruler left to anchor it.
4. Anything without a surface. Sky has no distance. A relative model maps it to the farthest value it can express — the number that means “as far as I know” and nothing more — and a metric model caps it at a dataset-dependent maximum. Any pipeline that converts “farthest” into “one kilometre” invents a measurement, so sky and other no-return pixels get masked.
Four places where a depth model is confidently wrong. None of them is a bug you can train away: each one deletes the cue the model needs.
Every row is a measurement the model cannot make from one picture, not a training-set gap. The fix is always additional information — a second view, another sensor, or a mask that tells the pipeline where not to trust the number.
The four failures share one shape: the model replaces missing information with its prior, smoothly and without complaint. That is also why the failures are invisible in a thumbnail — the map looks like a room. The practical response is never more training; it is more information — a second view, another sensor, a mask, or a known length — plus an honest report of what the map is allowed to claim.
A pixel is a direction; depth is a distance along it. Multiply one by the other and you have a point in 3D — the camera model that does it is the pinhole, and it needs exactly four numbers.
Think of the camera as a point with an image plane in front of it. Every pixel (u, v) defines one ray: go right by (u − cx) pixels, down by (v − cy) pixels, forward by f pixels, and you have a direction in camera coordinates. Scaling that direction until the forward component equals the depth Z gives the point:
The formula makes the camera’s sensitivity obvious. X and Y are proportional to (u − cx)/fx, so a 1% focal error is a 1% lateral error, while Z comes straight from the depth map and never touches the intrinsics. Compare two plausible guesses on this camera: f = 480 px (the truth) puts pixel (400, 340) at 3 m, giving X = 0.881 m; f = 448.6 px — the value a 60° assumption would use — gives X = 0.943 m. That is a 7% lateral error from a five-degree miscalibration, invisible in a thumbnail and fatal in a measurement.
The four numbers come from one of four places, in descending order of trust: the camera’s own API or EXIF metadata; a calibration target (a checkerboard photographed from several angles); a monocular intrinsics estimator that predicts the field of view from image content (Perspective Fields, UniDepth); or an assumption. The source is blunt about the last one — a 60–70° FOV guess is usable for visualisation, not for measurement. The cloud will look right and be wrong by a few percent, which is exactly the error mode this lesson keeps returning to.
import numpy as np
def depth_to_point_cloud(depth, intrinsics):
H, W = depth.shape
fx, fy, cx, cy = intrinsics
v, u = np.meshgrid(np.arange(H), np.arange(W), indexing="ij")
z = depth
x = (u - cx) * z / fx
y = (v - cy) * z / fy
return np.stack([x, y, z], axis=-1)
depth = np.random.uniform(0.5, 4.0, (240, 320))
pc = depth_to_point_cloud(depth, (320.0, 320.0, 160.0, 120.0))
print(pc.shape) # (240, 320, 3) — one 3-D point per pixel, in the camera frameEvery 3D lift is the same three lines: X = (u − cx)·Z/fx, Y = (v − cy)·Z/fy, Z = Z. Move the pixel or the depth and watch the point move; press lift to keep it.
The floor grid is drawn from the same formula run on the world side: project (X, 1.6 m, Z) into pixels and you get the floor. Lift enough pixels and the point cloud becomes the room — which is exactly what a real pipeline does with a predicted depth map and an intrinsics estimate.
Two details that save debugging time. The camera frame is x right, y down, z forward — y points down because image rows do, so a point above the optical axis has negative Y. And the depth map must be in the same frame: a network that predicts inverse depth or disparity needs one reciprocal before the lift, and a map produced on a centre crop needs the crop’s intrinsics, not the original frame’s. Both mistakes produce a cloud that looks like the scene but is subtly bent — the same failure signature as a wrong focal length. The points this chapter lifts are also what turns tracking metric: Lesson 27 takes each track’s box through this formula so identities are associated in metres, not just in pixels.
Depth evaluation looks simple and is not: the same prediction scores 0.0175 or 2.99 depending on whether you aligned it, and the wrong pixels can dominate the mean. Here is the protocol the papers actually use.
AbsRel — absolute relative error. For every valid pixel, divide the absolute error by the ground truth and average: mean(|d_pred − d_gt| / d_gt). Lower is better; 0.05–0.1 is the production band. Because the denominator is the truth, a 10 cm error at 1 m costs ten times as much as the same error at 10 m, which is the right weighting for most robotics and AR tasks: relative accuracy near the camera is what keeps you out of a wall.
δ < 1.25 — threshold accuracy. For every valid pixel, take max(pred/gt, gt/pred) and count it correct if the ratio is below 1.25 — within 25%. Higher is better, and the standard ladder reports three rungs: δ < 1.25, δ < 1.25² (1.5625) and δ < 1.25³ (1.953). State of the art sits at 0.9+ on the strict rung. The ladder matters: in the worked table below the one bad pixel fails the first rung and passes the second, and the pair of numbers together tell you it was a far outlier rather than a systematic bias.
Mask before you compute. Real depth datasets carry invalid pixels — sensor holes, saturated returns, missing ground truth — and both metrics divide by the ground truth, so a single zero explodes into a giant error term that swamps the mean. The source code takes a mask argument on both functions for exactly this reason, and the rule is: filter first, then measure. A sudden gap between your numbers and a paper’s is an evaluation bug far more often than a model bug.
The alignment step is the part beginners skip and reviewers check first. For a relative model, fit the single linear transform a · pred + b = target that minimises squared error over the whole image, then score the fitted prediction. Two sums of squares, no matrix library. What it removes is exactly the scale and offset the relative model never claimed to know: for a perfect map, a = 1/s and b = 0, and the bench shows raw AbsRel falling from 2.99 at a ×4 mis-scaling to 0.0145 after the fit. What it does not remove is any real error: the residual after alignment is the model’s own.
One frame, two worlds: a 1.2 m crate at 3 m, or a 4.8 m object at 12 m — drawn to their own scale, the pictures are identical. Pick a candidate and see what it does to the metrics.
This is why every MiDaS / Depth Anything paper reports metrics after alignment: without it you measure the arbitrary unit, not the model. It is also why relative depth cannot answer “how far is the wall?” — the same map fits the true room, a doll’s house, and a cathedral, pixel for pixel.
Your relative-depth model reports AbsRel 0.51 on a benchmark before alignment and 0.018 after. Which statement is right?
import torch
def abs_rel_error(pred, target, mask=None):
if mask is not None:
pred = pred[mask]
target = target[mask]
return (torch.abs(pred - target) / target.clamp(min=1e-6)).mean().item()
def delta_accuracy(pred, target, threshold=1.25, mask=None):
if mask is not None:
pred = pred[mask]
target = target[mask]
ratio = torch.maximum(pred / target.clamp(min=1e-6), target / pred.clamp(min=1e-6))
return (ratio < threshold).float().mean().item()
def align_scale_shift(pred, target, mask=None):
p = pred[mask] if mask is not None else pred.flatten()
t = target[mask] if mask is not None else target.flatten()
A = torch.stack([p, torch.ones_like(p)], dim=1)
coeffs, *_ = torch.linalg.lstsq(A, t.unsqueeze(-1))
a, b = coeffs[:2, 0]
return a * pred + bBefore leaving measurement, keep the two protocols apart in your head. A relative model is scored after alignment — shape only. A metric model is scored raw, because metres were part of the claim. Comparing a raw relative number with a raw metric number, or an aligned number with an unaligned one, is the most common way a depth table becomes meaningless.
The integration is five steps and one join. A detected box has pixels; the depth map has one number per pixel; lift the box through the depth and the 2D detection becomes a 3D point — the core of every AR occlusion system and every “pick up the cup” robot.
The pipeline the source describes is the one you will meet in production: (1) an RGB frame arrives; (2) a depth model produces the map; (3) a detector produces boxes; (4) each box’s centroid is lifted through the depth map to a 3D point and merged with the cloud; (5) the consumer acts — occlusion, planning, sizing, or replacing a stereo rig. The join in step 4 is the only new code, and it is the formula from the last chapter run once per box.
One depth value per box is enough for placement, and the choice of that value matters. A centroid is fine when the box is tight on one surface; a median over the box’s valid pixels is what you want in general, because a box on a person against a wall straddles two depths and the median lands on the object, not on the background the occlusion-edge failure bleeds in. The box’s size is the second free measurement: a 255-pixel-tall person box at depth 3.2 m with f = 480 px implies a real height of 255 × 3.2 / 480 = 1.70 m — which is f·H/Z run backwards, and a useful sanity check that the depth and the detector agree.
import numpy as np
def lift_box(box, depth, intrinsics):
x1, y1, x2, y2 = box
u, v = (x1 + x2) / 2, (y1 + y2) / 2
patch = depth[y1:y2, x1:x2]
valid = patch[(patch > 0) & np.isfinite(patch)]
z = float(np.median(valid)) # one depth value per box
fx, fy, cx, cy = intrinsics
X = (u - cx) * z / fx
Y = (v - cy) * z / fy
return np.array([X, Y, z]), z
point, z = lift_box((290, 150, 410, 405), depth_map, (480, 480, 259, 259))
print(point, z) # [0.607 0.123 3.2] 3.2The real-time arithmetic is worth doing before promising a frame rate. At 30 fps the whole budget is 33.3 ms. Depth Anything V2 Small at INT8 hits roughly 30 fps at 518 × 518 on a consumer GPU — the depth alone can spend the entire budget before the detector runs. In practice you choose: run depth at 10 Hz and interpolate, run both models on a smaller crop, quantise further, or accept a lower frame rate and let the planner work with the depth it has. The 2026 default — V3-style models on a GPU — makes the depth step cheap enough to be one stage among several, which is the real reason monocular depth changed what products are possible.
That closes the arc: a single frame goes in, a one-channel map comes out, four numbers lift it into geometry, and the evaluation protocol tells you how much of it to trust. The pieces you will reuse most are the smallest ones — the depth/relative distinction, the back-projection formula, the alignment step, and the habit of asking what the picture could not have contained.
Answer before you look. The relative-vs-metric question and the alignment question are the two that decide whether your next depth project produces a number or a unit — and the glass question is the one that tells you whether you trust the map too much.
0 / 6 answered · 0 correct
01What is the main difference between relative depth and metric depth?
02Depth Anything 3 (released 14 Nov 2025, ICLR 2026) uses a plain DINOv2 encoder plus a Dual-DPT head. What does its training recipe actually do?
03To lift a pixel (u, v) with predicted depth d to 3D, you use X = (u − cx)·d / fx, Y = (v − cy)·d / fy, Z = d. What are fx, fy, cx, cy?
04When evaluating a relative-depth model (MiDaS, Depth Anything), why do you apply scale-and-shift alignment before computing AbsRel?
05Your Depth Anything prediction on a glass-fronted reception desk reports believable but clearly wrong depth for the glass region. Why?
06Your own evaluation reports AbsRel 0.31 and δ < 1.25 = 0.41 where the model card claims 0.06 and 0.95, and you fed the metrics every pixel including zeros and NaNs. What should you fix first?
Click a card to swap the lazy description for what it actually means.
Three problems with exact numbers: find a failure in your own photos and name the cue that broke, lift a full frame to a point cloud with real and guessed intrinsics, and measure a known 30 cm move with a metric model — including the uncertainty arithmetic that says how many pairs you need. Try first; a worked answer is one click away.
The procedure is three lines with the Transformers depth pipeline — load, `pipe(image)['depth']`, convert from PIL to numpy — then normalise and save. What you will find, almost every time: the order is right and the geometry is local. Look for the reflective object (a monitor, a mug, a phone screen), the very close object at the frame's edge, or a uniform surface. A monitor is the reliable failure: the model reads the content on the screen as depth, so a video of a corridor reports a corridor — predicted 4–5 m for a screen 0.6 m away, an AbsRel above 1. The cue that failed is texture gradient: the screen's content has perspective and texture that contradict its physical flatness, and nothing in a single frame says which is real. The check that makes the exercise worth doing: hold a tape measure to the screen and compare with the model's value after a scale fit — the ordering survives, the metres do not.
Back-project every valid pixel: `X = (u − cx)·Z/fx`, `Y = (v − cy)·Z/fy`, `Z = d`, then stack into an (H·W, 3) array with the colours beside it. A 518 × 518 map is 268,324 points, which as float32 xyz is 268,324 × 12 B ≈ 3.2 MB plus 0.8 MB of uint8 colour — a `.ply` of roughly 4 MB, small enough to open on a laptop. With real intrinsics the room's floor comes out flat and walls meet at right angles; with a guessed 60° FOV (f = 259/tan 30° = 448.6 px instead of the true 480 px) the whole cloud is stretched laterally by 7% and the floor bends subtly — believable in isolation, wrong against a tape measure. The lesson's numbers: pixel (400, 340) at 3.00 m is (0.881, 0.506, 3.000) m with f = 480 and (0.943, 0.542, 3.000) m with f = 448.6. Ordering-quality clouds for visualisation, metric work for measured intrinsics — and if you cannot get them, estimate them from the image (Perspective Fields, UniDepth) instead of guessing.
Measure the object's distance in both frames by taking the median of the depth values over the bottle's pixels — never the centroid of a box, which mixes in the background. With a typical AbsRel of 0.06 at 2 m, one prediction carries ≈ 12 cm of error, so the 30 cm delta is a 12 cm signal in two 12 cm noises: a single pair can plausibly report anything from 0 to 50 cm. The uncertainty of a difference is √(σ₁² + σ₂²) ≈ 17 cm, so the signal-to-noise ratio on one pair is 30/17 ≈ 1.8 — detectable but not reliable. Average ten pairs and the noise on the mean falls to 17/√10 ≈ 5.4 cm, so the 30 cm delta is pinned to within about ±11 cm at two standard errors — ±36% of the effect, which is the kind of number a report can carry with the error bar attached. The lesson's three practical rules fall out: use a metric model if you have one, average many measurements when you do not, and quote an uncertainty with every delta — a delta from two single predictions has no error bar and therefore no meaning.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 04, Lesson 26) and the Math Foundations Notebook reference build. The five labs — the depth-map visualizer with its 1/Z hyperbola, the back-projection bench with the growing point cloud, the scale-ambiguity bench, the dated model comparator and the failure board — are original to this page, as is the arithmetic they compute: the depth-map bookkeeping (518 × 518 × 1 = 268,324 values = 1.07 MB float32 = 268 KB uint8, 8.05 MB/s at 30 fps), the stereo error law δZ = Z²δd/(f·B) at f = 480 px and a 6 cm baseline (28.80 px and 1.7% at 1 m, 2.88 px and 17.4% at 10 m, 0.29 px and 174% at 100 m), the patch arithmetic (518/14 = 37, so 1,369 tokens), the two hand-worked back-projections (pixel (400, 340) at 3.00 m → (0.881, 0.506, 3.000) m, range 3.167 m, 18.7° off axis; the 60°-FOV focal check at 448.6 px), the evaluation example (eight pixels → AbsRel 0.0875, δ<1.25 = 0.875), the alignment example (a = 2.000, b = 0.100, raw 0.513 → aligned 0.0175), the scale-bench numbers (raw 0.038/0.995/2.990/6.980 → aligned 0.0145), the detection lift (box (290, 150)–(410, 405) at 3.20 m → (0.607, 0.123, 3.200) m, 1.70 m tall and 0.80 m wide from the box) and the failure-board measurements (glass 2.40 m predicted at 5.80 m, a textureless wall drifting 2.70–5.30 m, a 4-pixel occlusion band at 3.50–6.00 m, an unmaskable sky). Every number shown is computed live by the labs or verified by hand in the prose.