3D vision comes in two flavours. Point clouds are the sensor’s raw output — N unordered points, no grid, no fixed N. NeRFs are the learned volumetric field — a network you can cast rays through to render a scene that no camera ever photographed. Both answer “what is where in space”, and choosing between them decides what the rest of your pipeline can do.
A point cloud, a mesh and a voxel grid store the geometry: you can loop over them, cut them, export them. An SDF and a NeRF store a function: ask for a coordinate and they answer. Ten centimetre voxels over a driving scene are 10⁸ cells = 400 MB per channel — which is why LIDAR stacks keep the raw points instead.
explicit lists · implicit answers · whose tool is next?A shared MLP runs on every point — the same weights, independently — and a max pool over the point axis leaves one fixed-size vector. Max is symmetric, so the output cannot depend on the input order, and it works for 8,000 points or 250,000. 807,626 parameters is the whole classifier; the pool itself has none.
f(P) = max over p in P of MLP(p)The network maps (x, y, z, direction) to (density, colour). A pixel is 64 coarse samples, 128 fine samples drawn from the coarse weights, and a transmittance-weighted sum: C = Σ Tᵢ(1 − exp(−σᵢδᵢ))cᵢ. Training is that loop against photographs — 20–50 posed views, no geometry, 1–2 days on one V100 for the original.
C = Σᵢ Tᵢ αᵢ cᵢ · αᵢ = 1 − exp(−σᵢδᵢ)A point cloud is the sensor’s set — N unordered points with no grid — and a NeRF is the network’s function — density and colour for every coordinate — and the one you pick is decided by what the next tool in the pipeline can consume.
By the end you will be able to sort any 3D task into explicit (cloud, mesh, voxel) or implicit (SDF, NeRF) and defend the choice with the representation’s cost; explain why PointNet’s shared MLP plus max pool is permutation-invariant and count its 807,626 parameters; trace a NeRF pixel from camera pose through 64 coarse and 128 fine samples to the transmittance-weighted composite; read the positional encoding (3 + 3 × 2 × 10 = 63 inputs, 512 cycles at the top band, 2 mm detail in a 1 m scene); compute Σw, the leftover transmittance and the depth for a ray by hand; and say out loud what a NeRF cannot give you — a mesh, a relightable material, or a static scene you can edit.
Every 3D pipeline answers “what is where in space” with one of five data structures. Three of them list the geometry; two of them only answer questions about it. That single difference decides which tools can consume the result.
A camera gives you a 2D image. A LIDAR gives you a set of 3D points with no ordering. A structure-from-motion pipeline gives you a sparse cloud of keypoints and a camera pose per photo. A NeRF gives you a network that can render a scene it has never seen a photo of. All four are “vision”, and none of them is the dense grid a CNN wants — so before any model, you have to choose a representation.
Explicit representations store the geometry. A point cloud is a list of N points. A mesh is a list of triangles with shared vertices. A voxel grid is a 3D image: a regular lattice of cells, each holding “occupied” or a feature vector. You can iterate over all three, count their elements, cut them, and export them.
Implicit representations store a function. A signed distance field (SDF) takes a point and returns the distance to the nearest surface, negative inside. A NeRF takes a point and a viewing direction and returns a density and a colour. Neither has a list of surfaces: the surface is where the function crosses zero (SDF) or where the density spikes (NeRF). You can ask them anything, but you cannot read them off.
The costs are not decoration, they are the decision. Voxelising a driving scene at 10 cm resolution over 100 m × 100 m × 10 m is 1000 × 1000 × 100 = 10⁸ cells, which is 400 MB per float32 channel — per frame, at 10 Hz, and almost all of it empty air. That single arithmetic is why autonomous-driving stacks keep the point cloud and run point-cloud networks on it instead of voxelising; voxel grids appear where the data already arrives that way (a CT scanner) or where the scene is small (a tabletop).
The same arithmetic runs the other way for capture. A NeRF’s entire scene is about 5 MB of network weights — two trunks at 596k float32 parameters each — where the paper’s local light-field baseline stores images and poses in the gigabytes (15 GB in its own comparison). Compression is what implicit representations are good at: the scene became a function, and a function is small.
Pick a task. The answer is decided by two things and nothing else: what the sensor hands you, and what the next tool in the pipeline needs.
| representation | explicit / implicit | you can ask it for |
|---|---|---|
| point cloud | explicit | PointNet-family nets, ICP registration, clustering, any LIDAR stack |
| triangle mesh | explicit | GPUs, Blender, game engines, CAD, physics solvers, 3D printers |
| voxel grid | explicit | 3-D convolutions, medical pipelines, occupancy prediction |
| signed distance field | implicit | collision checks, robotics planning, mesh extraction, level-set simulation |
| NeRF / Gaussian splat | implicit | novel-view renderers, viewers, thumbnails — not an editing or CAD tool |
The line that matters is not “old vs new”: it is whether the representation lists its geometry (point cloud, mesh, voxel) or answers questions about it (SDF, NeRF). Explicit formats are what editing tools eat; implicit ones are what sensors and gradient descent produce.
Two more pieces of vocabulary will come up all lesson. View dependence means the colour a point emits depends on the direction you look at it from — exactly what makes a NeRF render a glossy tabletop that brightens as the camera moves, and exactly what a textured mesh cannot do without extra material data. And posed images means photographs plus the camera position and orientation for each one: the one input radiance fields insist on, and the reason a structure-from-motion step runs before any training.
A point cloud is the raw output of every depth sensor: a list of N points in space with no grid, no connectivity, and no promise that the two clouds you compare have the same number of rows.
Written as a tensor, a cloud is (N, 3) — N points, three coordinates each — often with extra per-point channels like colour, intensity or a surface normal. A 64-beam LIDAR sweep is about 120,000 points; at 16 bytes per point (three float32 coordinates plus intensity) that is 1.9 MB per sweep, or 19 MB/s at 10 Hz. A depth camera gives tens of thousands of points per frame. A structure-from-motion reconstruction gives thousands of sparse keypoints.
None of these are a 2D image, and the reason is not the shape of the tensor — (N, 3) could be reshaped. The reason is that a convolution is defined by a grid: a kernel slides over a regular arrangement, and every output cell sees the same pattern of neighbours. A point cloud has no regular arrangement at all. Its neighbours are whichever points happen to be near in space, and that relationship changes with every sweep.
Two properties make a cloud awkward for a neural network, and they are worth separating because PointNet solves only one of them.
Why isn't 'sort the points, then feed them to a normal network' a fix for the unordered-input problem?
PointNet (Qi et al., 2017) made a neural network work on a raw point cloud with two lines of arithmetic: run the same small MLP on every point, then take the maximum down the point axis. Nothing in that operation can see the order the points arrived in.
The network is this:
Shared means what it says: one MLP, applied to every point independently, with no weights that depend on a point’s position in the list. In PyTorch it is implemented as a 1×1 convolution over the point axis (or a linear layer applied to the (N, 3) matrix), which is the same arithmetic in a form the GPU likes: Conv1d(3, 64, 1) treats the N points as a length-N “signal” with three channels.
Symmetric means the aggregation gives the same answer whatever order its inputs arrive in. A maximum is symmetric: max(a, b, c) = max(c, a, b). A sum is symmetric too, and both are used in the literature (max is the PointNet choice because it is robust to duplicate points and to outliers, and it keeps the strongest evidence for a feature rather than averaging it away). The result is one fixed-size vector — 1024 numbers — for a cloud of any size, in any order.
Take three points, a two-output shared MLP with weights W = [[2, −1, 0.5], [0, 1, 1]] and bias b = [0.5, −1], and ReLU after the linear layer. The same matrix is applied to all three points:
Two details to notice. The max runs down the N axis, not across the feature axis: each of the 1024 features gets the strongest response any single point produced. And the network needs no normalisation of N — a cloud of 16 points and a cloud of 250,000 points both leave the pool with a 1024-vector, because the pool discards the point axis entirely.
import torch
import torch.nn as nn
class PointNet(nn.Module):
def __init__(self, num_classes=10):
super().__init__()
self.mlp1 = nn.Sequential(
nn.Conv1d(3, 64, 1), nn.BatchNorm1d(64), nn.ReLU(inplace=True),
nn.Conv1d(64, 64, 1), nn.BatchNorm1d(64), nn.ReLU(inplace=True),
)
self.mlp2 = nn.Sequential(
nn.Conv1d(64, 128, 1), nn.BatchNorm1d(128), nn.ReLU(inplace=True),
nn.Conv1d(128, 1024, 1), nn.BatchNorm1d(1024), nn.ReLU(inplace=True),
)
self.head = nn.Sequential(
nn.Linear(1024, 512), nn.BatchNorm1d(512), nn.ReLU(inplace=True), nn.Dropout(0.3),
nn.Linear(512, 256), nn.BatchNorm1d(256), nn.ReLU(inplace=True), nn.Dropout(0.3),
nn.Linear(256, num_classes),
)
def forward(self, x):
# x: (batch, 3, num_points) — coordinates first for Conv1d
x = self.mlp1(x)
x = self.mlp2(x)
x = torch.max(x, dim=-1)[0] # (batch, 1024) — the symmetric function
return self.head(x)
pts = torch.randn(4, 3, 1024) # a batch of four 1,024-point clouds
net = PointNet(num_classes=10)
print(net(pts).shape) # torch.Size([4, 10])
print(sum(p.numel() for p in net.parameters())) # 807,626 — print it, do not guessWhere do the parameters live? Exactly in the four convolutions, their batch norms and the three-layer head — the symmetric pool has none, because a max has no weights to learn:
With 1,024 points per cloud that is tiny by modern standards — a ResNet-18 is 11.7M parameters — and it runs on a CPU in milliseconds. The descendants (PointNet++, Point Transformer, KPConv) add hierarchical sampling, local neighbourhoods and attention, but they all keep the two ingredients: per-point processing with shared weights, and a symmetric reduction to a fixed vector. Whenever you see a point-cloud network, look for those two lines first.
The same 16 points in a new order: watch the rows of the feature matrix change, the pooled max row stay put, and the label never move. That is the entire PointNet trick — a shared per-point MLP plus a symmetric function.
The head is a teaching model: distances to three reference clouds’ pooled features, not a learned classifier. The “straight line” cloud is outside those three categories, so its nearest reference is a guess — real PointNets learn the head and can only label what they were trained on, too.
NeRF (Mildenhall et al., 2020) answers “can a handful of photos become a 3D scene?” with a network that maps a point and a direction to a density and a colour. Rendering a new view is then a loop of ray casts through that network — and training is that loop run backwards.
Keep the interface in your head before any detail: the model is a function F(x, y, z, θ, φ) → (σ, r, g, b). Position and viewing direction in, density and colour out. Density σ says “how much stuff is at this point” — zero in empty air, large inside a surface. Colour is what that stuff looks like from that direction, which is why the direction is in the input: a specular highlight is the same point looking different from two angles.
There is no mesh, no voxel grid, no point list. Ask the network about any coordinate and it answers, whether or not a sample ever landed there before. That property — a continuous function rather than a stored array — is what makes the representation smooth, compact, and differentiable. It is also why the geometry is invisible: to know where a surface is you have to go and find the density.
Rendering one pixel of a new view is five steps:
t₁ … t_N between a near and a far plane. The paper uses 64 coarse samples spread over the whole interval.(σ, c). The coarse pass costs 64 queries per ray.That last table is worth pausing on, because it is the punchline of the whole representation. The scene — every surface, every colour, every glossy highlight — fits in 4.8 MB of weights, and those weights are the only thing you store. A single 800×800 photo is 1.9 MB raw; a hundred of them, plus their poses, is the 15 GB light-field baseline the paper compares against. The compression ratio is not a trick: the network is a compact function that happens to agree with the photographs.
Move the camera, pick a pixel, change the sample budget: the cross-section shows the pose and the samples, the panel shows the render, and the readout below is the composite arithmetic for that one pixel. Watch the running colour patch fill the ray from front to back.
Sample count buys accuracy here; the fine pass buys placement. This toy scene is smooth and only 3.4 units deep, so a uniform grid is already excellent — in a real room the ray crosses metres of empty air and the surface occupies a fraction of it, which is exactly why NeRF spends 128 of its 192 queries inside the coarse pass’s mass window.
import torch
import torch.nn as nn
def positional_encoding(x, L=10):
"""(..., D) -> (..., D * 2 * L): sin/cos of 2^l π x for l = 0…L−1."""
freqs = 2.0 ** torch.arange(L, dtype=x.dtype, device=x.device)
args = x.unsqueeze(-1) * freqs * 3.141592653589793
sinc = torch.cat([args.sin(), args.cos()], dim=-1)
return sinc.reshape(*x.shape[:-1], -1)
class TinyNeRF(nn.Module):
"""The real thing is 8 layers of width 256 with a skip; this is enough to
show the shape of the computation."""
def __init__(self, L_pos=10, L_dir=4, hidden=128):
super().__init__()
pos_dim, dir_dim = 3 * 2 * L_pos, 3 * 2 * L_dir
self.trunk = nn.Sequential(
nn.Linear(pos_dim, hidden), nn.ReLU(inplace=True),
nn.Linear(hidden, hidden), nn.ReLU(inplace=True),
nn.Linear(hidden, hidden), nn.ReLU(inplace=True),
nn.Linear(hidden, hidden), nn.ReLU(inplace=True),
)
self.sigma = nn.Linear(hidden, 1) # density: one number
self.color = nn.Sequential( # colour: view-dependent
nn.Linear(hidden + dir_dim, hidden // 2), nn.ReLU(inplace=True),
nn.Linear(hidden // 2, 3), nn.Sigmoid(),
)
def forward(self, x, d):
h = self.trunk(positional_encoding(x, 10))
sigma = torch.relu(self.sigma(h)).squeeze(-1) # σ ≥ 0
rgb = self.color(torch.cat([h, positional_encoding(d, 4)], dim=-1))
return sigma, rgb
nerf = TinyNeRF()
sigma, rgb = nerf(torch.randn(128, 3), torch.randn(128, 3))
print(sigma.shape, rgb.shape) # torch.Size([128]) torch.Size([128, 3])An 800×800 NeRF render uses 64 coarse and 128 fine samples per ray. Roughly how many forward passes through the network does one image need, and why does that number explain the original's ~30-second render time?
Feed a coordinate straight into an MLP and it can only build smooth shapes: ReLU networks are spectrally biased towards low frequencies. NeRF fixes it by lifting each coordinate into sin/cos waves at ten increasing frequencies before the first layer.
The fix is four lines of code and one idea: do not hand the network raw coordinates. Encode each coordinate x as
Why does this work? The claim is not that the MLP cannot represent a sharp edge — with huge weights it eventually can, and the paper’s ablation shows the no-encoding variant is not broken, it is mush: it scores several dB worse and its surfaces and textures dissolve. An MLP with ReLU units composes piecewise-linear functions, and to make a feature at frequency k out of the first layer you need weights of size about k. Frequencies that high are hard to reach, slow to learn, and unstable; frequencies that are already in the input cost one multiply.
The encoding also fixes the opposite problem on purpose: the raw coordinate is kept alongside its encoding (the + 3 in the sizes above), so the network gets both the smooth low-frequency signal and every octave above it. What it learns is which combination of bands belongs at which location, which is exactly the kind of thing gradient descent is good at.
The bands are octaves: 1, 2, 4, 8, … cycles across the scene. Frequencies between two bands are not handed over directly — the MLP composes them out of products (sin at 1× times sin at 8× contains 7× and 9×), which is why ten octaves are enough for a scene whose finest texture sits around 1/512 of its width. The practical floor is a little coarser than the top band suggests: two samples per cycle is the Nyquist limit, so detail below about 1/256 of the scene is where bands stop agreeing with each other. That is the honest version of “detail down to 2 mm in a 1 m scene” — 2 mm is the band, 4 mm is the comfortable resolution.
Normalisation matters for the same reason. The bands are fixed in normalised coordinates, so the whole scene has to be mapped into the cube first (NeRF scales the scene and the camera poses into a cube of side 2 around the origin). Capture in millimetres instead and every band above l = 0 wraps thousands of times per metre: the encoding becomes noise the network can neither use nor ignore. Scaling — not adding bands — is the fix, and the paper’s own ablation finds diminishing returns from raising the position bands past 10.
One coordinate, lifted into sin/cos bands: each band is twice as fast as the one below it. Add bands and the reconstruction of a scene profile sharpens — the residual is measured, not asserted.
The bands are octaves (1, 2, 4, 8, …), so a component at exactly 8× appears the moment L reaches 4 — and a component at 64× stays invisible until L reaches 7. Frequencies between two bands are not directly available: an MLP composes them out of products of the bands it was given, which is why 10 octaves are enough for a scene whose finest texture is around 1/512 of its extent.
You capture a 20 m wide outdoor scene with L = 10 position bands. What is the finest detail the encoding makes directly available to the MLP?
The rendering equation is five symbols long and it is the same one a CT scanner uses. NeRF’s contribution is not the equation — it is making it differentiable, so the density field that feeds it can be learned from photographs.
March along a ray and you pass through empty air (low density) and through surfaces (high density). Two quantities describe what happens: the opacity of a sample, and the transmittance — how much light is still left by the time the ray reaches it.
Read the four lines in order and the picture is simple. A sample’s opacity grows with its density and with its thickness δ — doubling either doubles the optical depth σδ. Transmittance is the product of everything that survived before it, so a dense surface early in the ray starves everything behind it. The weight is the two multiplied: a sample contributes only if light reached it and it absorbs something. Whatever transmittance is left at the end shows through as the background, and the inner product of the weights with the sample positions is the scene’s depth — a free by-product of the same arithmetic.
Just as important: this is differentiable. Every operation is a multiply, an exponential and a sum — no argmax, no sorting of surfaces, no discrete choice of which triangle a ray hits. That is what lets a photometric loss on a rendered pixel flow all the way back into the MLP’s weights, and it is the reason the whole representation is learnable.
Two properties of this arithmetic are worth noticing before you trust it. First, the weights look like a probability distribution over the ray — they are non-negative and they cannot exceed 1 in total — but the leftover is meaningful: Σw = 0.9502 says 5% of the light came from behind. A closed surface should give Σw → 1; if your NeRF reports Σw = 0.6 on a wall that should be solid, the density field is too thin in that region, and that is a training problem, not a rendering one. Second, the depth is a weighted average and inherits the same bias: where density ramps up gradually, the surface estimate sits in the middle of the ramp, slightly in front of or behind the true edge.
def volumetric_render(sigma, rgb, t_vals):
"""
sigma: (..., N_samples) density at each sample
rgb: (..., N_samples, 3) colour at each sample
t_vals: (N_samples,) distances along the ray
"""
# spacing between neighbours; the last sample gets a huge delta so it
# absorbs whatever is left (1e10 × anything = an infinite optical depth)
delta = torch.cat([t_vals[1:] - t_vals[:-1], torch.full_like(t_vals[:1], 1e10)])
alpha = 1.0 - torch.exp(-sigma * delta) # opacity
trans = torch.cumprod( # transmittance
torch.cat([torch.ones_like(alpha[..., :1]), 1.0 - alpha + 1e-10], dim=-1),
dim=-1,
)[..., :-1]
weights = alpha * trans # wᵢ = Tᵢ αᵢ
rendered = (weights.unsqueeze(-1) * rgb).sum(dim=-2) # the pixel
depth = (weights * t_vals).sum(dim=-1) # Σ wᵢ tᵢ
return rendered, depth, weights
t_vals = torch.linspace(2.0, 6.0, 64) # 64 coarse samples over 4 metres
sigma = torch.rand(64) * 0.5 # (a trained network supplies these)
rgb = torch.rand(64, 3)
rendered, depth, weights = volumetric_render(sigma, rgb, t_vals)
print(rendered.tolist(), depth.item(), weights.sum().item())The lab in chapter 4 prints this table for whichever pixel you pick, live. Set its sample budget to 16 coarse / 0 fine (the two sliders under the pixel controls) and its column and row to 24 / 16 — the centre pixel, whose ray goes through the big sphere only. The readout then gives Σw = 0.8106 on the sphere, a leftover of 0.1894, and a depth of 2.473 m. Now move to 27 / 20, where the ray clips the near sphere and the far one:
Change the sample budget and watch which numbers move — the table above is exactly what the lab prints as you drag the sliders. At 8 coarse samples the pixel colour is already within 1.25 levels of a 1,024-sample reference while the depth is off by 2.5 cm; at 64 coarse the depth is right to a millimetre. A colour is an average over the whole ray, so it is forgiving; a depth is a location, so it is not. That asymmetry is worth remembering the first time a NeRF render looks perfect while its extracted geometry is blobby.
A ray through your scene gives Σwᵢ = 0.83. What does the missing 0.17 mean, and what would you check?
Training a NeRF needs no 3D data at all — only posed photographs and a differentiable renderer. The limits come from the same place: what you get out is exactly what you put in, plus a network.
The training loop is the rendering loop run against a known answer. Sample rays through the camera that took photo k, composite each one into a pixel, compare with that photo’s pixel, take the squared error, backprop through the whole composite into the MLP. The paper’s loss is the L2 error summed over a batch of rays for both the coarse and the fine prediction — nothing else, no depth, no masks, no geometry:
“Posed” is the load-bearing word. The network never learns where a camera was — it is told, because a rendered ray has to be shot from the right place with the right direction to match a pixel in the photograph. That pose usually comes out of a structure-from-motion pipeline (COLMAP is the default), which also gives you a sparse point cloud for free. If the poses are wrong, the loss cannot go down no matter how long you train, and the symptom is a blurry blob rather than an error message.
The rest of the capture discipline is photographic rather than neural: keep the exposure and white balance fixed (a radiance field bakes in whatever brightness it saw from each direction, so auto-exposure becomes a shading artefact), avoid motion blur, get 60–80% overlap between neighbouring shots, and mask out anything that moves — a person walking through frame during a 20-minute capture becomes a ghost that no amount of training removes, because no single 3D scene explains those pixels.
# 0. install (one GPU, CUDA; see the docs for the exact wheels)
pip install nerfstudio
# 1. photos -> posed images. COLMAP runs here, and its sparse cloud is kept.
ns-process-data images --data ~/capture/raw --output-dir ~/capture/processed
# (~30 photos of an object, 60-80% overlap, fixed exposure, no motion blur)
# 2. fit the scene. The viewer streams while it trains.
ns-train nerfacto --data ~/capture/processed
# 3. look at it, from anywhere: the viewer runs during and after training.
ns-viewer --load-config outputs/.../config.yml
# 4. export. A mesh, a point cloud or a Gaussian-splat PLY, depending on the
# tool that has to consume it next.
ns-export poisson --load-config outputs/.../config.yml --output-dir exports/
ns-export gaussian-splat --load-config outputs/.../config.yml --output-dir exports/
# the 5-second version, for when you just want to see it work:
# instant-ngp's GUI: drag a data/nerf folder in, watch the fox appear.So what can a NeRF not do?
The lineage after 2020 attacked the first and fourth items in that list.instant-ngp (2022) replaced the MLP’s input with a multi-resolution hash grid and a tiny network, so the same loss trains in seconds to minutes — the repo’s headline is a NeRF of a fox in under five seconds on an RTX 3090. Mip-NeRF 360 fixed aliasing and unbounded scenes (a room plus everything outside the window). And 3D Gaussian splatting (2023) stopped treating the scene as a function at all: millions of explicit 3D Gaussians with opacity and view-dependent colour, rasterised instead of ray-marched, training in tens of minutes and rendering in real time at 1080p. Almost every radiance-field product in 2026 is splatting underneath; the NeRF vocabulary — radiance, transmittance, novel views, posed images — is what you use to think about it.
Pick the constraint your project actually has. The board reorders itself, and the row detail names what you give up by choosing it.
| method | kind | training | rendering | mesh export | storage |
|---|---|---|---|---|---|
| implicit | 1–2 days · 100–300k iterations on one V100 | ~30 s for one 800×800 image (123M MLP calls) | only by marching cubes on the density — coarse | ≈ 5 MB (two MLP trunks, 596k weights each) | |
| hybrid (implicit + a data structure) | the repo's headline: a fox in under 5 s on an RTX 3090 | interactive — tens of frames per second | yes, in the GUI (marching cubes on the hash grid) | hash grid + tiny MLP: tens to hundreds of MB | |
| hybrid (implicit + a data structure) | tens of minutes on a recent GPU (docs' V100 run: ~3 h for nerfacto-big) | interactive viewer during training and after | yes: ns-export poisson / marching cubes | hundreds of MB | |
| explicit | 30–60 min in the paper; a few minutes with the fastest variants | real time: ≥ 30 fps at 1080p, 100+ fps at lower resolutions | not natively — a separate extraction step | 0.5–5M Gaussians × 236 B = 118 MB–1.2 GB | |
| explicit | minutes to hours of CPU/GPU reconstruction | real time (the GPU's rasteriser, no learning) | yes, natively — a textured triangle mesh | megabytes | |
| explicit | minutes per scan; needs the hardware | real time | yes — registration and meshing are the whole pipeline | megabytes to gigabytes depending on density |
Sources: original NeRF paper (training, rendering, the ~5 MB scene), the instant-ngp repository (its 5-second fox headline), the nerfstudio documentation (its own V100 run and the capture advice), the 3D Gaussian Splatting paper (≥ 30 fps at 1080p, competitive training times), and the standard photogrammetry/scanning pipelines. Storage for 3DGS is computed from the paper’s representation: 59 float32 per Gaussian = 236 B.
The pattern behind the ranking: explicit representations are what other tools consume, implicit ones are what training produces. Production pipelines happily mix them — COLMAP’s sparse cloud seeds 3DGS, and a trained splat is often converted to a mesh right before it ships.
Answer before you look. The permutation question and the “client wants a mesh” question are the two that separate having read this chapter from being able to choose a representation in a kickoff meeting.
0 / 6 answered · 0 correct
01Why can't a plain CNN process a point cloud directly?
02What exactly makes PointNet permutation-invariant over the input points?
03A vanilla NeRF MLP fed raw (x, y, z) coordinates produces blurry results. What fixes it?
04How is a NeRF pixel computed?
05Your capture works, but the client needs to edit the asset in Blender and relight it. Why is a NeRF the wrong deliverable?
06Why has 3D Gaussian splatting largely replaced NeRF in production?
Click a card to swap the lazy description for what it actually means.
Three problems with exact numbers — prove permutation invariance and explain why sorting is not the fix, write a ray generator and check one pixel by hand, and train a TinyNeRF on a synthetic turntable with a reported PSNR. Try first; a worked answer is one click away.
Build the network from the lesson's code and compare: pts = torch.randn(1, 3, 512); idx = torch.randperm(512); shuffle the point axis (dim=-1 for the (N, 3, n_points) convention) and compare net(pts) with net(pts[:, :, idx]). Expect max|Δ| ≈ 1e-7 or exactly 0 — the arithmetic is the same max over the same 512 rows in a different order, so there is nothing else for floating point to disagree about. The max pool is the whole proof: max is commutative and associative, and every point passes through the identical MLP weights. A sorted-points baseline (sort the N points lexicographically by (x, y, z), then feed the sorted matrix to a CNN or an MLP) does make the input order-independent, but it fails as a shape descriptor: sort is discontinuous (a 1 mm nudge can swap two points and rewrite the whole ordering), it destroys locality — the first row and the last row are neighbours in space only by accident — and it defines an arbitrary convention rather than learning one. Symmetric aggregation is the version that survives contact with real scans.
Work in the camera frame first, then rotate. With a pinhole and focal length f, the direction for pixel (u, v) measured from the principal point is d_cam = ((u − cx)/f, (v − cy)/f, 1), and the ray is o = t, d = R · d_cam with (R, t) the camera-to-world pose — the same R and t the training images come with. Worked check: W = 800, H = 800, f = 1111 px (a ~40° horizontal field of view: 2·atan(400/1111) = 39.6°), pixel (700, 200). With cx = cy = 400: d_cam = ((700 − 400)/1111, (200 − 400)/1111, 1) = (0.270, −0.180, 1). Normalise if you want unit directions: |d| = sqrt(0.0729 + 0.0324 + 1) = 1.0516, so d̂ = (0.2568, −0.1712, 0.9509). With the pose R = Ry(30°) and t = (2, 0, 2) (a camera 2 m out at 30°, looking back at the origin), the world ray is o = (2, 0, 2) and d = Ry(30°)·(0.270, −0.180, 1) = (0.270·cos30 + 1·sin30, −0.180, −0.270·sin30 + 1·cos30) = (0.7338, −0.180, 0.7310). Sanity checks worth automating: the centre pixel (400, 400) must give exactly −t's direction after rotation (the camera looks at the origin by construction), the four corners must be the extreme directions, and a 90° field of view on a square image must give a half-angle of 45° — atan(400/1111) = 19.8° per side, i.e. 39.6° horizontal, which is the 36–40° window the NeRF datasets use.
The dataset is the hard part, and it is only 20 lines of code: place a cube at the origin, sample 20–50 evenly spaced camera poses on a circle at a fixed elevation (60° apart is 6 views; 100 views is a turntable), and for each pose render the cube with a brute-force ray tracer — for each pixel, intersect the ray with the 12 triangles (or with the 6 planes and keep the nearest front-facing hit), return the face colour, and stop. Store (RGB image, 4×4 pose) pairs; no depth, no mesh, no normals. Then train exactly as the lesson describes: sample 1,024 rays per step, for each ray take the 64 coarse samples, query (σ, c), composite, and take an L2 loss against the image pixel. Expect the coarse anatomy to appear within a few hundred steps (the mean colour of the scene), a recognisable cube by roughly step 1,000–5,000 (edges smeared, faces the right colours), and clean edges and specular-free faces by 10,000–50,000. Report PSNR, not raw L2, so the numbers are comparable to the literature: the synthetic Blender scenes in the paper reach 31.0 dB after 100–300k iterations on a V100, and a TinyNeRF at 40,000 steps on one modern GPU lands in the mid-20s dB — visibly right, visibly soft. Two traps to expect: no positional encoding means the cube never sharpens no matter how long you train, and a random 70/30 split of *rays* looks excellent while a held-out-pose split shows the blur, because neighbouring rays are almost the same ray.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 04, Lesson 13) and the Math Foundations Notebook reference build. The five labs (the canvas ray-casting visualizer with the live composite table, the canvas point-cloud playground, the representation chooser, the canvas positional-encoding explorer, and the NeRF-vs-alternatives board) are original to this page, as is the arithmetic they compute: the representation cost table (10⁸ voxels = 400 MB per channel; 8,000 points = 128 KB; a 4.8 MB NeRF), the exact PointNet parameter count (807,626, against the source comment's ~1.6M) with its layer table, the NeRF shape arithmetic (63 position inputs, 27 direction inputs, 283 colour-head inputs, 595,844 weights per trunk, 122.9M MLP calls for one 800×800 render, 786,432 samples per 4,096-ray batch), the two-band positional-encoding check (Δ at l=0 is 0.000314 versus 0.160676 at l=9 for a 0.0001 move), the volumetric-rendering worked example (σ = 0, 2, 4 at δ = 0.5 → α = 0, 0.6321, 0.8647, Σw = 0.9502 = 1 − e⁻³, depth 2.534 m) and the two-sphere pixel table (0.4497 / 0.3767 / 0.1736 at pixel 27, 20), and the 3D Gaussian storage arithmetic (59 float32 = 236 B per Gaussian). Every number shown is computed live by the labs or verified by hand in the prose.