Training Memory
A 70B model in BF16 is 141 GB of weights — and about 1.13 TB of training state before a single activation exists. This module derives all five terms from the update rule that requires them, then hands you a calculator to size real runs. It ends deliberately at the point where parameter sharding stops helping, which is where distributed training has to begin.
Core path ~31 min: 01 → 04 then 07 · Systems deep dive +24 min: 05 → 06
What problem existed?
After this beat you can: You can explain why a model that fits in memory still cannot be trained in memory.
A 70B-parameter model in BF16 is about 141 GB of weights. That already exceeds one 80 GB GPU, and inference people stop the story there.
Training does not. The weights turn out to be the smallest of the four persistent terms.
The reason is mixed precision. Forward and backward run in BF16 for speed, but the parameter update cannot.
Adding a small update to a large weight in BF16 loses the update entirely to rounding — BF16 has roughly 8 mantissa bits, so an update smaller than about 2⁻⁸ of the weight rounds away. The loss curve stalls.
So an FP32 master copy of every parameter is kept. And Adam keeps two more FP32 buffers per parameter for its moment estimates.
Each piece exists to prevent a specific failure under that recipe. This is the classic mixed-precision Adam accounting, not an invariant of training — low-bit optimizer states, different optimizer designs, and alternative precision schemes all change the constant.
Then there is a fifth term that behaves differently from the other four.
Activations are not parameter-scaled. You cannot get them from P. They are set by runtime shape and architecture: sequence length, micro-batch, depth, width, head count.
They are therefore the term you control at launch time, and the term that surprises people when a context-length experiment OOMs a run that trained fine yesterday with the same weights.
Getting this wrong is expensive in a specific way: you discover it minutes into a job that took an hour to schedule. Engineers who can size a run on paper first are doing arithmetic, not magic — and by the end of this module you will have done it for a 70B run yourself.
Two different scaling laws under one memory budget. Every lever touches one of them, not both.
How this comes up in interviews
Try answering out loud before revealing.
Which memory term cannot be computed from parameter count alone?
You have the mechanism. These teams had to decide with it under a production constraint — and you commit to an architecture before you are shown theirs.