0.1 + 0.2 = 0.30000000000000004 A float is a sign, an exponent and about seven significant digits. Everything past those digits is rounding — and the crashes of training live in the gaps.
One bit says which way, eight bits say how far, twenty-three bits say how precisely. The exponent buys range; the mantissa buys significant digits. float32 resolves about 7 decimal digits, float16 about 3.
value = (−1)ˢ × 2^(e−127) × 1.mantissafloat16 keeps 10 mantissa bits and tops out at 65,504. bfloat16 keeps float32's exponent range and only 7 mantissa bits. Training spikes past 65,504, so bfloat16 usually wins there; inference wants precision, so float16 wins there.
float16: precision · bfloat16: rangeexp overflows past 88.7 in float32 and 11.1 in float16. Shifting every logit by −max leaves the probabilities identical — the factor cancels — while the biggest exponent becomes e⁰ = 1, which always fits.
c + log Σ e^(x−c), c = max xA float is scientific notation with a fixed budget — one bit for direction, eight for scale, twenty-three for about seven digits — and every stability trick either keeps a result inside that budget (stable softmax, clamping, loss scaling) or rearranges the formula so the missing digits never matter (two-pass variance, factored differences, relative gradient checks).
By the end you will be able to read every dtype’s trade-off on sight, spot catastrophic cancellation before it ships, write stable softmax and log-sum-exp from memory, choose a gradient-check h that matches the precision of your code, and explain why bfloat16 trains without loss scaling.
IEEE-754 gives every float three fields: a sign, an exponent, and a mantissa. The exponent decides how far the number line reaches; the mantissa decides how finely it is marked.
A float32 stores a real number in 32 bits: one sign bit, eight exponent bits, and twenty-three mantissa bits. The value it represents is
Plain English: the exponent says how many places the binary point moves — that sets the range — and the mantissa records the leading digits — that sets the precision. In decimal scientific notation 6.02 × 10²³, the mantissa is 6.02 and the exponent is 23; here the base is 2 instead of 10 and the budget is fixed. The leading 1 is implicit, so float32 really has 24 bits of significant information, about 7 decimal digits.
Numbers that are sums of powers of two — 6.5, 2.25, 0.125 — fit exactly. Everything else is rounded to the nearest representable float. The gap between neighbouring floats is called one ulp (unit in the last place), and it is what makes floating point leaky.
Type any number. See its float32 bits, where it sits between its neighbours, and what float16 and bfloat16 would store instead.
The gap is the value’s whole resolution: add anything smaller than half of it and the sum rounds back to where it started. Near 1.0 that gap is 1.19e−7 for float32 — machine epsilon.
0 10000001 10100000000000000000000.1.0 + x ≠ 1.0: the machine epsilon.float32 stores 6.5 exactly but only approximates 0.1. Which field of the float decides that?
Exponent bits buy range; mantissa bits buy precision. The 16-bit formats split those 16 bits in opposite ways — and that choice is why one is for training and the other for inference.
float32 and float64 are the everyday formats. float16 and bfloat16 halve the memory and speed up matmul on Tensor Cores, but each must give something up: float16 trades range for mantissa bits, bfloat16 trades mantissa bits for range. The source table, with exact extremes:
| Format | Bits | Exp. | Mantissa | Digits | Largest | Smallest normal |
|---|---|---|---|---|---|---|
| float64 | 64 | 11 | 52 | ~16 | 1.8e308 | 2.2e−308 |
| float32 | 32 | 8 | 23 | ~7 | 3.4e38 | 1.2e−38 |
| bfloat16 | 16 | 8 | 7 | ~2–3 | 3.4e38 | 1.2e−38 |
| float16 | 16 | 5 | 10 | ~3 | 65,504 | 6.1e−5 |
Move one value across 650 decades. Each format’s bar is its representable range; red is overflow to inf, blue is underflow to zero. The small tick marks the gap at 1.0 — its precision.
Range and precision are separate budgets: bfloat16 spends its 16 bits on float32’s exponent range and keeps only 7 mantissa bits; float16 does the opposite. Training needs range first.
Rounding is the everyday symptom. The number 0.1 is a repeating fraction in binary, so it cannot be stored exactly at any finite precision. The nearest float64 is 0.1000000000000000055511…, and the nearest float32 is 0.100000001490116119384765625. Add the stored values and the error that was invisible surfaces:
Plain English: every decimal you type is first snapped to the nearest float, and arithmetic then happens on the snapped values. The fix is not a better formula — it is to compare with a tolerance and to expect drift when millions of tiny terms accumulate.
One bit of mantissa resolves a factor of two; in decimal that is a factor of log₁₀ 2 ≈ 0.30103. Multiply by the number of bits:
The exponent budget works the same way: float32’s 8 exponent bits span 2⁻¹²⁶ to 2¹²⁷, roughly 1.2e−38 to 3.4e38; float16’s 5 bits span 2⁻¹⁴ to 2¹⁵, or 6.1e−5 to 65,504. Drop below the smallest normal and you enter the subnormal range, where precision degrades; drop below the smallest subnormal (5.96e−8 for float16) and the value is zero.
In Python (float64), why is 0.1 + 0.2 == 0.3 false?
When two nearly equal numbers are subtracted, the digits they share vanish and the rounding noise hiding in their last bits becomes the answer. The error never grew — the result shrank to the size of the error.
Every stored float carries a small absolute error, roughly its size times machine epsilon. If a subtraction cancels the leading digits, that error stays the same size while the result becomes tiny — so the relative error explodes. The textbook example, in float32:
One subtraction, a 19% error, and no exception raised. The same pattern appears in three places every ML engineer meets: computing a variance as E[x²] − E[x]², applying the quadratic formula when b² ≫ 4ac, and choosing too small a step h for a finite-difference gradient.
Subtraction deletes shared digits. Two classic casualties: the variance formula E[x²] − E[x]², and a² − b² when a and b are close.
The fix is never “more digits” — it is rearranging the formula so the large, nearly equal numbers never meet in a subtraction.
For data with a large mean, the naive formula squares numbers that are all nearly equal. The squares have only ~7 significant digits in float32, so their difference — the variance — is swamped. The source example, [1000000, 1000001, 1000002], has population variance exactly 2/3 = 0.6667:
The quadratic formula has the same disease. For x² − 1000000x + 1 = 0, the roots are near 1e6 and 1e−6; subtracting the nearly equal 1e6 from √(1e12 − 4) destroys the small root. Since the product of the roots is the constant term, compute the small root as C / (large root):
You subtract two float32 numbers that agree in their first six digits. What happens to the relative error of the result?
Overflow turns a result into infinity, underflow turns it into zero, and undefined operations turn it into NaN. From there, one bad value can poison every later calculation.
The boundaries are exact, not fuzzy. A float32 can hold 3.4028234663852886e38, the largest value below 2¹²⁸; anything larger rounds to inf. Its smallest positive value is the subnormal 1.401298464324817e−45; anything smaller than half of that (below 2⁻¹⁵⁰ ≈ 7.006e−46) rounds to 0.0. Taking the natural log of the overflow point gives the exp() domain limit:
Plain English: exp amplifies size, so it is the main source of overflow in ML — softmax, sigmoid, and probability math are full of it. log runs the other direction: its input must be strictly positive, and an underflowed 0 becomes −inf. The dangerous composition is log(exp(x)) without a shift.
NaN (Not a Number) comes from undefined operations: 0.0 / 0.0, inf − inf, inf × 0, √(−1), log(−1). Infinity comes from overflow or division by zero. Both propagate: any arithmetic touching a NaN produces NaN, and one NaN gradient makes its weight NaN, which makes every later output NaN. Training dies within a step. Detection is cheap — math.isnan, math.isinf, math.isfinite — and worth running after every forward pass while debugging.
Softmax exponentiates logits and divides by their sum. The division is what makes the answer scale-free — and that same scale-freedom is the loophole that makes the computation safe.
Softmax turns K real-valued logits into K probabilities that sum to 1:
Plain English: exponentiate to make every score positive and to exaggerate the gaps, then divide by the total so the results are proportions. Subtracting the same number m from every logit multiplies numerator and denominator by e^(−m), and the factor cancels — the probabilities are mathematically identical. Choose m = max(z) and the biggest exponent becomes e⁰ = 1, which cannot overflow, while the sum is at least 1, so the divisor is never 0.
The lesson’s source values: with logits [2, 1, 0.1] both routes give [0.659, 0.242, 0.099]. With logits [100, 101, 102] the naive route computes e¹⁰⁰ = inf in float32 and returns [nan, nan, nan]; the stable route shifts to [−2, −1, 0], gets exponentials [0.135, 0.368, 1.000] summing to 1.503, and returns [0.090, 0.245, 0.665] — exactly what a perfect implementation should.
Add a growing offset to every logit [2, 1, 0.1]. Shifting all logits must not change the probabilities — the naive code only disagrees because eᶻ left the format’s range.
In float32, eᶻ overflows once z passes ≈ 88.7. Subtracting the max makes the largest exponent 0, and exp(0) = 1 always fits.
Both facts follow from the exponent rule e^(a+b) = e^a · e^b. For the shift:
Log-sum-exp is the same trick seen through a logarithm. It is the quantity inside every log-probability and every cross-entropy:
The stable form is not an optimization; it is a requirement for correctness. Frameworks fuse softmax and cross-entropy so that the loss is computed from logits with log-sum-exp inside.
import math
def softmax_stable(z):
m = max(z) # largest logit
e = [math.exp(v - m) for v in z] # largest exponent is 0
s = sum(e)
return [x / s for x in e]
def logsumexp(z):
c = max(z) # factor out e^c
return c + math.log(sum(math.exp(v - c) for v in z))
def cross_entropy_stable(true_class, z):
return logsumexp(z) - z[true_class] # −log softmax(z)[t]Edit up to three values. The naive route exponentiates immediately; the stable route subtracts the max first. Watch which one survives.
Your stable softmax subtracts max(logits) before exp. The largest exponent is 0. Why does that guarantee no overflow?
Gradient checks and normalization layers both take a tiny constant — a step h, an epsilon ε — and both have a sweet spot. Push either one in the wrong direction and the stability they promise turns into the bug.
Lesson 05 introduced the centered difference (f(x+h) − f(x−h)) / (2h) as an independent referee for backpropagation. This chapter asks the numerical question behind it: which h? Two errors pull in opposite directions. Shrink h and the secant hugs the tangent — the truncation error falls like h². But shrink h too far and f(x+h) and f(x−h) agree in almost every digit, so their subtraction is catastrophic cancellation — the roundoff error grows like ε/h. The best h sits where they cross, and that crossing depends on the format.
Plain English: every evaluation of f is rounded to the working precision. The difference f(x+h) − f(x−h) is supposed to be about 2h·f′(x); once rounding error in the two evaluations is comparable to that difference, dividing by 2h amplifies pure noise. The source recommends h = 1e−5 to 1e−7, which is right for float64; in float32 the table above puts the sweet spot nearer 5e−3.
The centered difference from Lesson 05, now with the precision in charge. Sweep h across nine decades and watch truncation error and roundoff error meet at a sweet spot.
The best h depends on the format: this sweep bottoms out near 6e−3 for float32 and near 3e−6 for float64. Lesson 05’s advice of 1e−5 to 1e−7 is tuned for float64.
Expand both evaluations with Taylor series (Lesson 04) around x:
The check compares analytical and numerical gradients with a relative error so that scale does not matter: |a − n| / max(|a|, |n|, 1e−8). Below 1e−7 is excellent, below 1e−5 is acceptable, above 1e−3 there is a bug, and above 1 the gradient is simply wrong. This also explains a real debugging rule: match the precision of the check to the precision of the code. Running a float32 model’s gradients through a check with h = 1e−8 manufactures failures out of rounding noise.
Normalization layers use the same kind of tiny constant for the same kind of reason. LayerNorm recenters and rescales the features of each example; BatchNorm does the same across the batch. The source writes the layer as (x − mean) / (std + ε) · γ + β; PyTorch keeps the epsilon under the square root, (x − mean) / √(var + ε) · γ + β. Either way ε exists so that all-identical activations do not divide by zero:
The learned γ and β let the network restore any scale it needs, so the layer can be gentle. And because LayerNorm and BatchNorm are exactly the places where variance is computed, they are also where a two-pass or Welford statistic matters in half precision. Frameworks compute those statistics in float32 even when the activations are float16.
Tensor Cores multiply float16 matrices several times faster than float32. Mixed precision takes the speed but keeps a float32 master copy of every weight — and loss scaling keeps the smallest gradients alive.
Pure float16 training breaks in two places at once: activations above 65,504 overflow, and gradients below about 3e−8 underflow to zero. The recipe that works, and that every automatic-mixed-precision (AMP) implementation follows, is:
Derivatives are linear — a constant multiplies out of the whole chain (Lesson 04): d(S·L)/dw = S · dL/dw. So scaling the loss scales every gradient in the network by exactly S, and dividing afterwards recovers the true gradient in a format that can hold it.
bfloat16 needs none of this for range: it keeps float32’s 8 exponent bits, so its smallest positive value is about 9.2e−41 and its largest is 3.39e38. It pays in precision — only 7 mantissa bits, about 2–3 decimal digits — but training tolerates that far better than it tolerates zeroed gradients. float16 keeps 10 mantissa bits and is preferred for inference, where values are bounded.
Gradients can also be too large. Exploding gradients from deep stacks — the eigenvalue story of Lesson 03, compounded — can corrupt every weight in a single step. The standard fix is clipping by norm: if the gradient vector is longer than c, rescale it to length c while keeping its direction.
Clip by value is the simpler alternative — clamp each entry separately — but it can rotate the gradient direction, which clip by norm provably preserves. Normalization layers are the other half of the safety net: keeping activations recentered every layer stops the ten-fold growth that would hit inf by layer 50.
Why is bfloat16 usually preferred over float16 for training, even though it has fewer mantissa bits?
| Symptom | Likely cause | Fix |
|---|---|---|
| Loss becomes NaN | softmax overflow, or learning rate too high | stable softmax, lower lr, clip gradients |
| Loss stuck at log(K) | uniform outputs: dead ReLUs or wrong labels | LeakyReLU or GELU, check data and loss |
| Accuracy 1–3% below the paper | float16 without loss scaling | dynamic loss scaling (AMP), or bfloat16 |
| Some layers’ gradients exactly 0 | dead ReLUs or float16 underflow | as above; check initialization |
| Different results on different GPUs | non-deterministic summation order | accept ~1e−6, or force deterministic algorithms |
| exp() returns inf in the loss | raw logits into exp | use log_softmax / log-sum-exp |
Answer before you look. The softmax and bfloat16 questions are the ones that show up in real training runs — and the exercises are all recomputed in exact IEEE-754 values.
0 / 5 answered · 0 correct
01What is the approximate range of numbers that float32 can represent?
02Why does 0.1 + 0.2 not equal 0.3 in floating-point arithmetic?
03In the stable softmax implementation, why subtract max(logits) before exponentiating?
04When using centered finite differences for gradient checking, what happens if the step size h is too small (e.g., 1e-15)?
05Why is bfloat16 generally preferred over float16 for neural network training?
Click a card to swap the lazy description for what it actually means.
Four short problems from the source lesson, worked end to end. The 3×2 layer gradient check is the one-dimensional version of the lab in chapter 06.
Naive in float32: the mean is exactly 1000001. Each square is rounded first: 1000000² → 999999995904, 1000001² → 1000002027520, 1000002² → 1000003993600. Their average also rounds to 1000002027520, and mean² = f32(1000001²) = 1000002027520, so E[x²] − E[x]² = 0 — a 100% error, even though the true variance is 0.6667. Two-pass: subtract the mean first (deviations −1, 0, 1, all exact), square (1, 0, 1), average → 0.6666666865348816, a relative error of 3.0e-8. Welford's online update gives the same: n=1 → M2=0; x=1000001, delta=1, mean=1000000.5, M2=0.5; x=1000002, delta=1.5, mean=1000001, M2 = 0.5 + 1.5·1 = 2.0, so variance = M2/n = 2/3 = 0.6667 (or M2/(n−1) = 1.0 for the sample variance). The fix is to subtract the big number before squaring, so the squares never see the mean.
Floats near 1.0 have exponent 0, so consecutive floats are one unit in the last mantissa bit apart: 2⁻²³ = 1.1920928955078125e-7. Adding exactly half a step, 2⁻²⁴ = 5.9604645e-8, is a tie and rounds to even — back to 1.0. So the smallest x with 1 + x ≠ 1 is 2⁻²³, and numpy.finfo(numpy.float32).eps returns 1.1920929e-07, the same number. Numeric check: 1.0 + 2⁻²³ = 1.0000001192092896, while 1.0 + 2⁻²⁴ = 1.0. The same recipe gives 2⁻¹⁰ = 9.765625e-4 for float16 and 2⁻⁵² ≈ 2.22e-16 for float64.
(a) [4, 4, 4]: naive = ln(e⁴+e⁴+e⁴) = ln 3 + 4 = 5.098612; stable = 4 + ln(3·e⁰) = the same. Both fine. (b) [0, −800, −900]: in float64 e^−800 underflows to exactly 0, so naive = ln 1 = 0 and stable = 0 + ln(1 + 0 + 0) = 0 — both fine, because the dominant term carries the answer. (c) [−1000, −1001, −1002]: every exponential underflows to 0, so naive = ln 0 = −inf. Stable shifts by c = −1000: terms 1 + e⁻¹ + e⁻² = 1.5032147, so the answer is −1000 + ln(1.5032147) = −1000 + 0.407606 = −999.592394. The naive version overflows at the top too: [500, 501, 502] gives every eˣ beyond float32's 3.4e38 maximum, so it returns inf, while the stable version returns 502 + ln(1 + e⁻¹ + e⁻²) = 502.407606.
float16's smallest positive value is 2⁻²⁴ = 5.96e-8, and anything below half of that (2⁻²⁵ = 2.98e-8) rounds to zero. Take six gradients [3e-9, 2e-8, 5e-8, 6e-8, 4e-7, 1e-6]: the first two are below the 2.98e-8 halfway line, so 2/6 ≈ 33% are flushed to 0. Multiply every gradient by 1024 first: [3.07e-6, 2.05e-5, 5.12e-5, 6.14e-5, 4.10e-4, 1.02e-3], all at or above the denormal floor, so 0% become zero. After backward, divide the scaled gradients by 1024 in float32: 3.07e-6 / 1024 = 2.998e-9, recovering the update that float16 alone had erased. The gradient still carries only float16's ~3 digits, which is why the master weights stay in float32.
You do not need to master these here. Each one gets a proper treatment in its own lesson; the one-line meaning is enough to keep reading. Orange dotted underlines in the prose point back to this list.
This lesson is a port of an open course. Everything here traces back to it — and the next step is running the code yourself.
Lesson text adapted from AI Engineering from Scratch (Phase 01, Lesson 13) and the Math Foundations Notebook reference build. Float inspector, precision ranger, cancellation bench, softmax/log-sum-exp labs and the gradient-check sweep are original to this page, and every displayed value is rounded exactly as IEEE-754 would round it. All labs run in your browser.