A scene is a cloud.
Render it by rasterising.

Start with what the previous lesson’s NeRF actually costs. A rendered pixel is a sum over samples along one ray: 64 coarse plus 128 fine samples, each one an MLP forward pass for a density and a colour. An 800 × 800 image is 640,000 rays, so one frame is 640,000 × 192 = 122,880,000 MLP evaluations — call it 123 million network calls for one picture. Training runs 100,000–300,000 iterations of that, which is why the original NeRF took one to two days on a V100 and rendered at roughly a frame every 30 seconds.

3DGS asks a different question. If the scene is already a cloud of primitives, why integrate along rays at all? Project each primitive to the screen, sort by depth, and composite — the operation a GPU rasteriser was designed for. The 2023 paper renders 1–5M Gaussians at 134–154 fps on an RTX A6000, with every scene holding above 30 fps at 1080p. The follow-up research and the tooling around it settled the numbers everybody quotes in 2026: 10–30 minutes to train a scene on an RTX 4090, and a scene that renders in real time on hardware you already own.

The second difference is less obvious and more consequential. A NeRF has no parts. If you want to move a chair inside a trained scene, there is no “chair” — the appearance is entangled across the network’s weights, so the only honest answer is to retrain. A 3DGS scene is a list: each Gaussian has its own position, size, orientation, opacity and colour, and the pixels it affects are exactly the pixels it covers. Pick a subset and translate it and you have moved the chair. That single property — editable primitives — is why 3DGS became the production default, and why the industry standardised it in 2026: Khronos ratified the KHR_gaussian_splatting glTF extension (release candidate February 2026) and OpenUSD 26.03 shipped the UsdVolParticleField3DGaussianSplat schema.

What did not change is why either method looks right. Both compute the classical volumetric render — transmittance-weighted sums of colour along a view ray. NeRF evaluates it with dense samples through an MLP; 3DGS evaluates it with sparse explicit Gaussians and a rasteriser. Same equation, same photometric loss, different physics of computation. That identity is why their final image quality lands in the same band, and why the honest comparison table below is about speed, storage and editability rather than pixels.

The axes where the difference is real — and the one where it is not (neither representation is a mesh) — read better as a panel than in prose. Pick a scene size and watch which rows separate the two representations:

One number from the NeRF lesson is worth carrying forward because it explains the whole trade: the entire NeRF scene was about 5 MB of float32 weights — two trunks of 596k parameters each. A 3DGS scene of one million Gaussians, at the 59 floats per Gaussian this lesson derives in chapter 02, is 236 MB. Two orders of magnitude more memory buys two orders of magnitude more speed and the ability to edit the result. As you will see, that trade has a cost of its own, and it is measured in gigabytes and in artefacts called floaters.

A Gaussian is a blob, not a point. What makes it a blob is that it is anisotropic — it can be long and thin in one direction and squat in another — and oriented, so the long direction points somewhere specific in the scene. Two of the six attributes exist for that reason alone: a rotation and a per-axis scale. Together they build the 3 × 3 covariance that says “how far from the centre does this Gaussian still matter, along each direction?”

Read the ladder. Degree 0 keeps one number per channel — a constant colour, the average of what the Gaussian looks like from everywhere. Degree 1 adds a linear ramp: the colour can differ between the top and the bottom, like a diffuse surface under a single light. Degree 3 costs 48 of the 59 floats and buys enough angular resolution for Lambertian shading, a specular highlight, and mild reflection. That is the 3DGS default, and it is why the colour is 4.4× the geometry: 48 / 11.

The two activations in the list are not decoration. Scales are stored as logarithms and exponentiated at render time, because a scale is a positive quantity and gradient descent has no idea about positivity: in log space every real number is a valid scale, and an additive step of 0.1 multiplies the actual size by e^0.1 ≈ 1.105 — a 10.5% nudge, whether the Gaussian is 1 cm or 1 m across. Opacity is stored as a logit and squashed by a sigmoid, for the same reason: the optimiser gets the whole real line, the renderer gets [0, 1]. Colour is squashed by a sigmoid too.

Now spend the budget on a realistic scene. A room captured from 40 photos typically converges to about 2 million Gaussians. Raw float32 storage is 2,000,000 × 236 B = 472,000,000 B = 472 MB. Quantised to the 32-byte .splat layout it is 64 MB — a seventh of the size, with the SH terms gone. And training is the part that actually hurts: four copies of every parameter (the value, its gradient, and Adam’s first and second moment) is 472 MB × 4 = 1.89 GB, before the rasteriser has allocated a single per-tile buffer. That number is why the training batch size and the densification schedule are the two settings people tune when a run runs out of memory.

The centre is easy: the world position goes through the camera transform and lands at a pixel. The shape is the interesting part, because perspective is not linear — things shrink as they get farther away, and they shear as they move off-axis. The standard approximation replaces the true warp with its first-order behaviour at the Gaussian’s centre: a 2 × 3 Jacobian matrix, evaluated once per Gaussian, that turns a small 3D displacement into a small 2D one.

Run one Gaussian through by hand — this is the lab’s default state, so every number below is on screen. Take μ = (0.5, 0.25, 4), focal length f = 800 px, axis scales σ = (0.4, 0.15, 0.05) in metres and no rotation. The covariance is Σ = diag(0.16, 0.0225, 0.0025). The Jacobian at μ is J = [[200, 0, −25], [0, 200, −12.5]] — the 200s are f/z = 800/4, and the −25 and −12.5 are the perspective shear terms −f·x/z² and −f·y/z². Multiply through:

Two lessons hide in those lines. First, the projected size is just the focal length times the Gaussian’s size divided by depth — f·σ/z, the same arithmetic a pinhole camera has been doing since 1850, and the check that catches a wrong Jacobian immediately. Second, the tiny z axis contributes 0.02% of the diagonal but all of the off-diagonal: shear is a small number that changes the ellipse’s orientation, not its size. An axis-angle readout alone would have looked identical; only the covariance knows.

Now rotate the splat 45° about its own y axis — the lab’s “foreshortening” preset. The long 0.4 m axis now points halfway between the image plane and the view direction, and the projection reflects it:

The last ingredient is the falloff. Every pixel inside the ellipse receives α = σ · exp(−½ dᵀΣ′⁻¹d) — a weight that is 1.0 at the centre and decays with the squared Mahalanobis distance. That decay is a Gaussian, so a fraction of the total mass lands inside each contour, and the fractions are universal:

That last row is the reason a tile-based rasteriser can be fast. A splat’s contribution at 3σ is e^(−4.5) = 0.0111, and an 8-bit pixel cannot represent a difference smaller than 1/255 = 0.0039. Discard every Gaussian whose α at a pixel is below 1/255 and the pixel is wrong by less than one unit of colour — which is why real implementations cull at a threshold instead of evaluating a million tails.

The key move is to stop thinking about rays. A NeRF asks “what is along this ray?” and walks the ray. A splat renderer asks “which primitives can this pixel see?” and gathers them. Gathering is what rasterisers do, and it maps onto the GPU’s execution model without translation.

1. Project. Every Gaussian’s centre and covariance go through Σ′ = J W Σ Wᵀ Jᵀ (chapter 03), producing a 2D ellipse and a depth — the centre’s distance along the camera’s viewing axis.
2. Bin. The screen is cut into 16 × 16 pixel tiles. A 1080p frame is 1920 × 1080, i.e. 120 × 68 = 8160 tiles. Each Gaussian is appended to every tile its ellipse overlaps — a cheap 2D bounding-box test.
3. Sort. Each tile sorts its own list by depth, nearest first. The sort is per tile, not global: two splats at different depths never need a definite order unless they share a pixel.
4. Composite. Each pixel walks its tile’s list, accumulating colour and transmittance — the blend below.
5. Write. The accumulated colour and opacity replace the pixel. Nothing about this step knows or cares that the primitives were Gaussians.

Why tiles at all? Locality. A tile is small enough that its entire working set fits in registers and shared memory, so the per-pixel loop never touches global memory. And because splats overlap, a Gaussian that covers a 80 × 30 pixel ellipse is already counted in ceil(80/16) × ceil(30/16) = 5 × 2 = 10 tiles — the tiling turns one big primitive into ten small, cacheable jobs. The cost of the tiling is one list entry per (splat, tile) pair, which is the rasteriser’s real per-frame working set; the parameter store in chapter 02 is the persistent cost.

Three splats at one pixel, with α = 0.6, 0.5, 0.2 in front-to-back order, does the whole thing on one line of arithmetic per splat. The first arrives with full transmittance and contributes 0.6; the second arrives with T = 1 − 0.6 = 0.4 and contributes 0.5 × 0.4 = 0.2; the third arrives with T = 0.4 × 0.5 = 0.2 and contributes 0.2 × 0.2 = 0.04. Total opacity A = 0.6 + 0.2 + 0.04 = 0.84, and the residual 1 − 0.84 = 0.16 is exactly (1 − 0.6)(1 − 0.5)(1 − 0.2) — the light that made it through the stack to the background. With red, green and blue colours for the three splats the premultiplied pixel comes out (0.60, 0.20, 0.04), and the lab prints the same table for whatever pixel you click.

Look closely at that equation and you will recognise the previous lesson. NeRF’s volumetric render composites C = Σ Tᵢαᵢcᵢ over samples along a ray; 3DGS composites the identical expression over explicit primitives. The difference is the sampling: NeRF’s samples are dense and implicit (192 per ray, chosen by a coarse pass), 3DGS’s are sparse and explicit (only the Gaussians that actually cover the pixel, already sorted). Nothing in the physics changed, which is why the two land in the same quality band — and why every intuition you have about transmittance carries over.

One more property makes the whole training story work: every step above is differentiable with respect to the Gaussian parameters. The projection is a matrix product, α is an exponential of a quadratic form, the composite is a sum of products, and the SH evaluation (chapter 05) is a dot product. Given a photograph, take the difference in pixel space, backpropagate through the rasteriser, and update (μ, q, s, α, c_lm) — all 59 floats per Gaussian. There is no sampling step to differentiate around, no ray-marching discretisation: the renderer is the network.

Walk around a real object and its appearance changes: the diffuse part stays roughly steady, the specular part slides across the surface, and metal and varnish do not behave the same way as paper. A single RGB value per Gaussian cannot express any of that — every Gaussian would look like flat paint. The cheapest fix that fits in the same data-oriented renderer is to store the colour as a function of direction, in a basis that is smooth, cheap to evaluate and small enough to keep per Gaussian.

That basis is the spherical harmonics: the Fourier series of the sphere. Basis function Y_lm is indexed by a degree l and an order m, and for each degree there are 2l + 1 orders, so everything up to degree L is 1 + 3 + 5 + … + (2L+1) = (L+1)² functions. Degree 0 is a constant; degree 1 is a linear ramp (a dipole); degree 2 is quadratic; degree 3 — the 3DGS default — is (3 + 1)² = 16 functions per channel. Three channels, so 16 × 3 = 48 floats. That is the entire mechanism: learn one coefficient per basis function per channel, and evaluate a dot product at render time.

Evaluation is one dot product per channel. At degree 1 the basis is short enough to write out in full, and it is worth doing once so the numbers are not a mystery. With d = (x, y, z) the unit view direction and c = (c₀, c₁, c₂, c₃) the learned coefficients for one channel:

Notice the sigmoid in the third column. 3DGS squashes the evaluated value into [0, 1] before it becomes a colour, exactly as it squashes opacity: the coefficients live on the whole real line and the optimiser is free to move them anywhere. If you want to see a three-channel version of the same arithmetic, the lab’s viewing direction is a unit vector and every channel has its own 16 coefficients — the code below is the full degree-3 evaluation from the lesson’s reference implementation.

The loop is the same one the previous lesson established: render, compare, backpropagate. What changes is what the gradient reaches. In a NeRF it lands on the MLP’s weights; here it lands on 59 floats per Gaussian, and the number of Gaussians is itself an output of training. The loss is the reference implementation’s composite of an absolute-error term and a structural-similarity term:

Why two loss terms? L1 is robust and direct — it measures the absolute colour error and does not punish a few large mistakes as brutally as L2 does — but it is nearly blind to structure: a blurred render and a sharp one can have similar per-pixel error. SSIM compares local windows (mean, variance, covariance) and is high when edges line up, so the 0.2 term pushes the cloud toward crisp structure while the 0.8 term keeps the colours honest. The pairing is not unique to 3DGS; it is the standard recipe in the NeRF literature too.

Densification is the mechanism that lets a fixed-size seed become a scene. Each Gaussian carries a gradient magnitude for its position — the training signal’s opinion about how badly it needs to move. Where that magnitude is high, the region is under-reconstructed, and the fix depends on the Gaussian’s current size:

• Clone — high gradient, small scale. The Gaussian is in the right place but cannot resolve the detail alone, so duplicate it and let the copy take a share of the region.
• Split — high gradient, large scale. One big soft blob is covering structure it cannot represent, so replace it with two smaller Gaussians positioned along its long axis.
• Prune — opacity has decayed below 0.005, or the Gaussian has grown oversized in world space and no longer contributes meaningfully. Remove it and its parameters.

The growth is dramatic and it is the honest cost of the method. A run that starts with 100,000 seeded Gaussians commonly finishes with 1–5 million — a 10–50× increase in the parameter store over the run. In bytes, using the 236-byte record from chapter 02: the seed is 100,000 × 236 B = 23.6 MB, and the final scene is 236 MB to 5,000,000 × 236 B = 1.18 GB. Add the optimiser’s four copies and the training peak moves from 94 MB to between 944 MB and 4.72 GB. That is why 3DGS runs are described in terms of VRAM as much as in minutes, and why the same scene trained at a lower SH degree is a different engineering decision.

The practical workflow has not changed much since the paper’s release, and that is a good sign: the community converged on a pipeline instead of a stack of incompatible research scripts. Shoot the scene. Recover the camera poses. Train. Clean. Export. View.

pip install nerfstudio gsplat

# 1. structure from motion: run COLMAP, write the poses + sparse cloud
ns-process-data images --data photos/ --output-dir data/

# 2. train: splatfacto is nerfstudio's 3DGS trainer, 30k iterations
ns-train splatfacto --data data/            # 10-30 min on an RTX 4090
#    the viewer streams the run while it trains

# 3. export the Gaussians (and a mesh, if you need geometry for physics)
ns-export gaussian-splat --load-config outputs/.../config.yml --output-dir exports/

# 4. optional: clean and quantise in a browser editor, then serve
#    SuperSplat -> .splat (32 B/splat) -> any glTF/Three.js viewer

Capture advice, in the order it matters. Photograph a static scene — anything that moves ruins the multi-view geometry. Keep 60–80% overlap between neighbouring views, and shoot an arc or a spiral rather than orbiting from one spot so that every surface is seen from several angles. Lock exposure, aperture and focus if you can: auto-exposure makes the same surface a different colour in different images and the optimiser will faithfully bake that inconsistency into the SH. 20–50 photographs is the working range for an object or a room; an outdoor scene is 100–300. Matte, textured surfaces reconstruct best; glass, chrome, plain white walls and foliage are the hard cases.

Which tool? The differences are small in 2026 and you can change your mind later, because the asset is a list, not a network. In practice: gsplat (the CUDA rasteriser and training kernels); nerfstudio · splatfacto (the end-to-end recipe); Inria reference implementation (the original 2023 code); SuperSplat (the editor); PostShot / SplatForge (commercial front ends). The rasteriser matters when you train, the editor matters when you ship, and the export format matters forever.

The honest limits

Memory grows with detail. Storage is the paper’s own headline limitation. A room can be 2M Gaussians (472 MB raw, 64 MB quantised) and a street 5M (1.18 GB raw) before anyone complains about fidelity; training adds a 4× multiplier. There is no quality dial that does not move bytes.

Floaters. Gaussians in empty space look like dust hanging in the air, and they are usually evidence rather than error: the optimiser put opacity where the photographs could not disagree with it. They are removed by hand in an editor, or reduced by better capture, or both. Counting them is a useful quality metric: a scene with hundreds of floaters has an under-constrained region somewhere, and moving the camera near it will show you where.

No collision geometry. A cloud of translucent blobs has no surface, no inside and no volume you can query robustly. A physics engine, a path tracer and a CNC machine all need something a splat is not. Hybrid pipelines extract a surface — from the same capture with photogrammetry, or from a depth model — and keep the splats for appearance. Plan for both deliverables if a simulator is in the loop.

Baked lighting. As chapter 05 warned, SH stores what you saw, so relighting means re-rendering with new appearance models, which mostly means research code. If the client says “put it in the engine and light it with our sun”, the answer is a mesh, not a splat.

Resolution and aliasing. The projection is a first-order approximation (chapter 03), so a splat seen very close, or a scene rendered far below its capture resolution, can shimmer or pop. Mip-Splatting-style filtering and keeping splats at a sane size both help. Check any asset at the extremes of the framings you intend to ship, not only in the orbit the demo uses.