Field Guide
GPU Capacity Planner · Field Guide

How the planner actually works

Every number the planner shows you, explained in plain language — so you can trust it, argue with it, and catch it when it's wrong. Part one is inference. Part two is training and fine-tuning. Part three is how those two connect to the cost model, and why they're one tool now instead of three. Part four covers what's been added most recently.

26 chapters No prior GPU knowledge assumed ~30 min read
01

Two separate questions

Everything the tool does comes down to answering two independent questions about a model you want to run:

1. Does it fit? The model itself, plus every conversation currently happening, has to physically sit in the GPU's memory at the same time. If it doesn't fit, nothing else matters — the system can't even start.

2. Is it fast enough? Once it fits, can it produce answers quickly enough for however many people are using it, given how fast you've promised they'll get a reply?

Think of a restaurant kitchen. Question 1 is "does the walk-in fridge hold enough ingredients for tonight's bookings?" Question 2 is "can the cooks actually plate that many dishes before people get annoyed waiting?" A kitchen can fail on either one separately — plenty of fridge space with one slow cook, or a fast brigade with nowhere to store the food.
Two paths, one answer Model + KV cache Concurrency + speed target Does it fit? → GPUs Fast enough? → GPUs max( both ) = answer
Fitting and speed are graded separately. The tool never adds them.

The tool answers both questions separately, then takes whichever one needs more GPUs — because you have to clear both bars, not just one.

02

Vocabulary, before anything else

Ten words carry the whole tool. Once these click, the rest is arithmetic.

Parameters — "8B", "70B", the model's size
A model is built from billions of adjustable numbers it learned during training — its parameters. "70B" means 70 billion of them. More parameters generally means a more capable model, but also a bigger one to store and move around — the single biggest lever on everything downstream.
Precision — BF16, FP8, FP4
Each of those billions of numbers has to be stored somehow, and precision is how much space each one takes. BF16 uses 2 bytes per number (the common default). FP8 halves that to 1 byte. The 4-bit formats (NVFP4, MXFP4, INT4) take about half a byte — slightly more than exactly half, because small blocks of numbers share a "scale" value that has to be stored too. Lower precision means a smaller, faster model that's very slightly less sharp — usually a good trade. Many new models now ship in FP8 or FP4, which the tool calls "native."
Why it matters: this one setting alone can cut your GPU requirement in half or more.
VRAM — GPU memory
The dedicated, very fast memory built into the GPU chip itself — separate from your computer's regular RAM. The model's numbers, and every live conversation, have to live here while it's "thinking." An H100 has 80GB; if the model plus its conversations need more than that, you need more than one GPU.
Token — the model's unit of text
Models don't read whole words — they read tokens, chunks of text roughly ¾ of a word each. "500 tokens" is about 375 words, or a solid paragraph. Every input you send and every word the model writes back is counted in tokens — it's the currency the whole tool runs on.
KV cache — the conversation's short-term memory
While the model is writing a reply, it has to keep track of everything said so far — your question, and everything it's already written — so the next word stays on-topic. That running memory is the KV cache, and critically: every single ongoing conversation needs its own copy. This is the number people forget to budget for.
Why it matters: 200 people chatting at once means 200 separate slices of this memory, all resident at the same time — often bigger, combined, than the model itself.
Tensor parallelism (TP) — splitting one model across GPUs
When a model is too big to fit on one GPU, you cut it into pieces — say, quarters — and put each quarter on its own GPU. Those GPUs then work as a team, all contributing to producing every single word. A model split across 4 GPUs this way is running with "TP=4."
Replica — one working copy of the whole setup
However many GPUs it takes (via tensor parallelism) to hold one working copy of the model — that team, together, is one replica. If one replica can't keep up with demand, you don't make it bigger — you make more replicas and split the traffic between them, the same way a busy shop opens a second till instead of hiring a superhuman cashier.
Throughput — tokens per second, in total
How many tokens the whole system can produce per second, added up across every conversation happening at once. This is the "speed" side of sizing, separate from whether things fit — a system can have plenty of memory and still be too slow.
Batching — serving many people off the same GPUs at once
Modern serving software doesn't handle one person, finish, then move to the next — it works on many people's requests together in a batch. Each step reads the whole model from memory once and produces one word for everyone in the batch, so bigger batches are dramatically more efficient. The catch: each step gets a little slower as the batch grows, so every individual person's reply gets slower too. The tool finds the biggest batch that still keeps each person's reply fast enough.
Why it matters: how close real software gets to the hardware's theoretical speed — the serving efficiency (default 50%) — is the assumption most likely to be wrong. Measure your own stack and calibrate it (Chapter 22).
Concurrency — how many people, right now
The number of separate conversations happening at the exact same moment — not total users, just simultaneous ones. 5,000 people might use the app in a day, but only 200 might be actively mid-conversation at any given second. Sizing is done against the second number.

Precision has a wrinkle worth understanding on its own: how a model gets squeezed down to a lower precision affects how much accuracy survives the trip — but not the memory math this tool does.

Naive rounding breaks down on large models because of outliers — roughly 0.1% of a model's internal values run up to 20× larger than everything around them, and a rounding grid built wide enough to fit those few extreme values wastes almost all its resolution on the other 99.9%. Five well-known techniques each solve this outlier problem at a different point in the pipeline:

RTN — round-to-nearest
Ignores the outlier problem entirely. Rounds every weight straight to the nearest grid value, no calibration data needed. Cheapest to run, weakest at low bit-widths — accuracy falls off a cliff below about 4 bits.
GPTQ — repair after rounding
Quantizes a layer column by column, then uses each column's leftover error to nudge the not-yet-quantized columns, so mistakes get compensated for instead of compounding down the layer.
AWQ — protect before rounding
Finds the ~1% of weight channels that matter most — the ones driving the largest activations — and scales them up before quantizing, so they survive the rounding grid intact. Everything still ends up at the same bit-width.
LLM.int8() — isolate the outliers
Splits the outlier ~0.1% of dimensions out to run in FP16 at inference time, runs the other 99.9% in INT8, then merges the two results back together.
QAT — quantization-aware training
Solves it during training rather than after the fact: the model is fine-tuned with rounding baked into every forward pass, so it adapts to the precision loss before quantization is ever applied for real.
Why this tool only asks for a bit-width, not a method

All five techniques above produce the same artifact: a model stored at a given bytes-per-parameter, whichever one made it there. They differ in how much accuracy survives the trip — not in how much memory the result takes up. That's the whole reason this tool's precision input is just a format (BF16 / FP8 / NVFP4 / MXFP4 / INT4) with no method picker alongside it: memory footprint is method-agnostic, accuracy is not, and sizing only needs the former. (Source: a comprehensive survey of these methods at arXiv:2411.02530.)

Same outlier, five different fixes ~0.1% of values run ~20× larger than the rest · each method repairs it at a different stage OUTLIER, ~20× one weight matrix's values (illustrative) — a rounding grid sized to fit this one spike wastes resolution on everything else TRAIN CALIBRATE ROUND DEPLOY RTN GPTQ AWQ LLM.int8() QAT
Every method reaches the same bytes-per-parameter result. They differ only in which stage of the pipeline repairs the outliers — filled dot marks where.
Tensor parallelism, then replicas One replica = TP=4 team · three replicas serve three times the traffic REPLICA A GPU 1/4 GPU 2/4 GPU 3/4 GPU 4/4 REPLICA B GPU 1/4 GPU 2/4 GPU 3/4 GPU 4/4 REPLICA C GPU 1/4 GPU 2/4 GPU 3/4 GPU 4/4 traffic splits evenly across replicas A, B, C
12 GPUs total: 3 replicas × 4 GPUs each, not one team of 12.
03

Question 1, worked: does it fit?

Using the tool's own default scenario — Llama 3.1 70B, BF16, 200 concurrent chats, 500 tokens in / 500 tokens out, a reply inside 10 seconds, on H100s — here's what each row of the "Memory Constraint" table actually means:

Memory constraint — reading the tool's own numbers as plain English
NumberWhat it meansValue
Model weights70.6 billion numbers × 2 bytes each (BF16) = the model itself, installed.141.2 GB
KV cache, one conversationThe short-term memory for a single 1,000-token exchange (500 in + 500 out).0.33 GB
KV cache, all conversationsThat 0.33 GB, times 200 people chatting simultaneously.65.5 GB
Safety marginBuffer for the serving software's own runtime, ~10% of the above.20.7 GB
Total memory neededEverything above, added up — what has to fit somewhere.227.4 GB
Usable space per GPUAn H100's 80GB, minus ~10% the system always reserves for itself.72 GB
→ Smallest team that fitsTeams come in sizes of 1, 2, 4 or 8 GPUs (the model's attention heads have to split evenly). 4 H100s hold the weights with 33 GB per GPU left over for conversations — room for about 368 of them.4
In one sentence

A 70-billion-parameter model fits on a team of 4 H100s with room to spare for its conversations — but fitting is only the first question, and here it isn't the one that decides the answer.

04

Question 2, worked: is it fast enough?

A 500-token reply inside 10 seconds means every person needs at least 50 tokens per second streamed back to them. Each decoding step has to read every weight from memory, so a team's speed for a single person is capped by how fast it can read the model:

Throughput constraint — reading the tool's own numbers as plain English
NumberWhat it meansValue
Each person needs500 tokens ÷ 10 seconds.50 tok/s
4-GPU team, one conversationThe fastest a 4-GPU team can stream to a single person, at 50% serving efficiency. Below 50 — so the team that fits is too slow, no matter how few people use it.38 tok/s
8-GPU team, one conversationTwice the GPUs reading the model in parallel. Not quite twice as fast — every step also pays a fixed cost for the 8 GPUs to sync up — but there's headroom above 50.71 tok/s
Largest batch that stays fast enoughThe tool grows the batch until each person's speed drops to just above 50 tok/s.93 people
The 8-GPU team, serving that batch93 people × ~50 tok/s each — about 584 tok/s per GPU. Still limited by memory bandwidth, not compute.4,672 tok/s
What 200 people actually need200 people × 50 tok/s.10,000 tok/s
→ Teams needed200 people ÷ 93 per team, rounded up.3
In one sentence

4 GPUs can hold the model, but only 8 can stream it fast enough for one person — and 200 people need 3 of those 8-GPU teams, so the final answer is 24 GPUs.

05

Why "the bigger number," not "add them up"

Question 1 said "a team of 4 fits." Question 2 said "a team of 8 is needed for speed, and 3 of them." The tool takes the team size that satisfies both — 8 — and then asks how many of those teams the traffic needs: 8 × 3 = 24. Fitting and speed aren't stacked costs; the speed answer is "how many teams," and each team is already at least the size that fits.

If your use case had been 10 people instead of 200, one 8-GPU team would have been enough — the answer would stay at 8 GPUs, because one team was always going to serve them all. And if you'd asked for a 5-second reply instead of 10, each person would need 100 tok/s — more than even a single 8-GPU H100 team can deliver to one person. The tool flags that as not reachable rather than pretending more GPUs will fix it: you need faster GPUs, lower precision, or a looser target.

Lower precision moves both questions at once. Switch the same scenario to FP8 and the weights halve, so each step reads half as much: a single 8-GPU team now streams 128 tok/s to one person and can serve all 200 at once. The answer drops to 8 GPUs — about three times the tokens per GPU.

06

What's a fact, what's a guess

Not every number on the page carries the same weight. Some come from real, published hardware and model specifications. Others are the tool's best estimate of something genuinely uncertain — and you should treat those as a starting point to test, not a promise.

Verified

  • Model size (params) — publicly documented per model
  • Model's internal shape (layers, memory-per-token) — from the model's own published config
  • GPU memory size (VRAM) — a spec sheet number
  • GPU memory speed (bandwidth) — a spec sheet number
  • Bytes-per-number at each precision — a fixed fact of BF16/FP8/FP4, including the 4-bit formats' shared scales
  • Memory footprint at a given precision, regardless of quantization method (RTN/GPTQ/AWQ/LLM.int8()/QAT) — see Chapter 2

Assumption

  • Serving efficiency — how close your software gets to the hardware's theoretical speed (default 50%); the biggest wildcard by far, so calibrate it from your own stack
  • Pipeline efficiency — the cost of splitting a model across nodes (default 90% per hop)
  • Tensor-parallel efficiency — real-world team coordination is never perfectly linear
  • Usable VRAM % — how much the runtime quietly reserves for itself
  • Framework overhead % — a rough buffer, not a measured number
  • Your own traffic pattern — real usage is bursty, not a smooth constant
Rule of thumb

Trust the left column as-is. Treat every number in the right column as a slider to stress-test — run the tool once optimistic, once conservative, and see how much the answer actually moves. If it swings wildly, that's the assumption worth validating with a real benchmark before you buy anything.

PART 2Training & fine-tuning — teaching the model something new
07

A different kind of question

Inference answers questions live, one exchange at a time, forever. Training is the opposite shape: one long, continuous run that has to get through a fixed amount of material once, then stop. The two questions that mattered for inference still apply, but they mean something different:

1. Does one copy fit? Before training can even start, the model plus everything the training process itself needs to remember has to sit in memory together — and that "everything else" turns out to be much bigger than the model alone.

2. How long will it take, and how many GPUs does that mean? Given the total amount of work the job requires, and a deadline you're willing to accept, how many GPUs do you need running together to get there in time?

If inference was a kitchen serving diners in real time, training is a factory with a fixed order to fill by Friday — the question isn't "can we plate fast enough right now," it's "will the ovens we have get every loaf baked before the deadline."
08

Vocabulary, part two

A few new words, and one you already know wearing a different hat.

Gradient — the correction signal
After the model attempts an answer and gets scored against the right one, every trainable number gets told which direction to nudge to do better next time. That instruction is its gradient. Every parameter being trained needs one, the same size as the parameter itself.
Optimizer state — the training algorithm's own memory
The standard training algorithm (Adam) doesn't just apply each correction blindly — it keeps a running memory of recent corrections for every trainable parameter, so adjustments stay smooth instead of jerky. That memory has to be stored too, and for Adam it's the single biggest line item in the whole memory budget — bigger than the model itself.
Why it matters: weights + gradients + optimizer state together run to roughly 16 bytes per parameter for standard mixed-precision training — 8x the 2 bytes/param the model alone takes up at the precision used for inference.
Full fine-tuning vs. LoRA — train everything, or train a little
Full fine-tuning adjusts every single parameter in the model — maximum flexibility, maximum memory cost (that 16 bytes/param applies to the whole thing). LoRA freezes the entire base model and bolts on a small set of new, adjustable parameters (often under 1% of the total) — only those need gradients and optimizer state, so the memory bill shrinks dramatically.
Activations & activation checkpointing — the model's scratch work
While computing an answer, the model produces a huge amount of intermediate scratch-work at every layer, and normally has to keep all of it around until the correction pass is done. Activation checkpointing throws most of that scratch-work away as it goes and recomputes it on demand later — the exact reduction depends on model shape, but it's typically a large one (often 5-40x), paid for with a modest amount of extra compute.
FLOPs — the unit of "how much math"
Short for floating-point operations — the basic unit of computational work. Where inference sizing was measured in tokens per second, a training job is sized by its total FLOPs: a fixed, known amount of arithmetic that has to happen once, from start to finish, no matter how many GPUs you throw at it.
MFU — Model FLOPs Utilization
A GPU's spec sheet lists a theoretical top speed that real training never actually sustains — communication between GPUs, waiting on data, and imperfect scheduling all eat into it. MFU is the fraction of that advertised peak you actually achieve. This is training's version of serving efficiency from Part 1: the number most likely to make or break a real estimate.
Model-parallel vs. data-parallel — splitting the model, vs. copying it
Model-parallel is the same idea as tensor parallelism from Part 1: cut one model into pieces across several GPUs because it doesn't fit on one. Data-parallel is different — run several complete copies of the model side by side, each chewing through a different slice of the training data at the same time, purely to finish the whole job faster.
Full fine-tuning vs. LoRA, memory footprint FULL FINE-TUNE LoRA Layer 1 · grad + optim Layer 2 · grad + optim Layer 3 · grad + optim ... every layer Layer 1 · frozen Layer 2 · frozen Layer 3 · frozen + adapter · grad + optim every layer costs 16 bytes/param frozen layers cost 2 bytes/param only
Same forward pass touches every layer either way — only the trainable slice keeps gradients and optimizer state.
09

Question 1, worked: does one copy fit?

Same model as before — Llama 3.1 70B — but now full fine-tuning on a 2-billion-token dataset:

Memory — reading the tool's own numbers as plain English
NumberWhat it meansValue
Weights + gradients + optimizer state70 billion parameters × ~16 bytes each — the model, its correction signals, and the optimizer's own memory, all at once.1.12 TB
Activation memoryScratch-work for a batch of 2 sequences at 4,096 tokens each, with full activation recomputation (checkpointing) enabled.10.7 GB
Safety marginBuffer for the training framework's own runtime, ~10% of the above.113.1 GB
Total memory neededEverything above, added up.1.24 TB
Usable space per GPUAn H100's 80GB, minus the runtime's own reserve.72 GB
→ GPUs needed to fit one copy1.24 TB ÷ 72 GB, rounded up.18
In one sentence

Fully fine-tuning a 70B model needs 18 GPUs working together just to have somewhere to put the model, its corrections, and the optimizer's memory — before asking how long the job takes.

10

Why LoRA changes the answer so much

Same model, same 2-billion-token job — but train only the small LoRA slice (~1% of parameters) instead of the whole thing:

Full fine-tuning vs. LoRA — identical job, same GPU model
MetricFull fine-tuningLoRA
Memory needed1.24 TB178 GB
GPUs to fit one copy183
Total training compute8.4 × 10²⁰ FLOPs5.6 × 10²⁰ FLOPs
Total GPUs recommended183
Why the compute row only moves a little

LoRA's compute row is lower, but not proportionally lower — about ⅔ of full fine-tuning's, not ~1% even though only ~1% of parameters are being trained. Every forward pass still touches all 70B parameters, frozen or not. In the backward pass, every parameter — frozen or trainable — still needs its input-gradient computed, since that's the signal relayed further back through the network to eventually reach the trainable adapters. What frozen parameters skip is their own weight-gradient — the update instruction, which only a trainable parameter needs. Skipping that halves the backward-pass cost for the ~99% of parameters that are frozen, which is where the savings come from. LoRA is mostly a memory trick with a real, smaller compute saving on top — memory drops by ~7x here (1.24TB → 178GB) while compute drops by only ~1.5x, which is why the GPU count is dominated by the memory constraint (3, not something compute-derived) at this dataset size.

11

Question 2, worked: how long, and how many

Back to full fine-tuning's 18 GPUs. How long does the actual job take, and does it need more GPUs than the fitting question alone required?

Compute — reading the tool's own numbers as plain English
NumberWhat it meansValue
Total training compute6 × 70 billion parameters × 2 billion tokens — the standard estimate for a full forward-and-backward pass over the whole dataset.8.4 × 10²⁰ FLOPs
Per-GPU achievable speedAn H100's spec-sheet peak, discounted to the 40% of it real training actually sustains.396 TFLOP/s
Target training timeThe deadline: what you told the tool you're willing to wait.14 days
→ Extra copies needed to hit that deadlineThe 18 GPUs already required just to fit turn out to be so much combined power that they clear the 14-day target easily — no extra copies needed.1
Actual time at 18 GPUsFar under budget — a side effect of needing that many GPUs just to fit, not because speed demanded it.1.5 days

That's the opposite of Part 1's example, where speed forced extra GPUs beyond what fitting alone required. Here it's flipped — full fine-tuning at this scale is so memory-hungry that fitting alone gives you far more speed than you asked for. To see the other direction, tighten LoRA's deadline from 14 days to 2 days, keeping everything else the same:

LoRA, same job, tighter deadline — now speed is the constraint
Metric14-day deadline2-day deadline
GPUs to fit one copy33
Data-parallel copies needed14
Total GPUs312
Actual time achieved6.1 days1.5 days
What changed

Same 3-GPU team, just run 4 identical copies side by side, each working through a different quarter of the dataset at the same time, then combining what they learned. That's data parallelism — the training equivalent of Part 1's "open more replicas" move.

12

What's a fact, what's a guess — training edition

Same discipline as Part 1. Some numbers are load-bearing facts; others are the tool's best estimate of something genuinely contested.

Verified

  • Model size and shape — published per model
  • The ~16 bytes/parameter mixed-precision Adam figure — a standard, well-documented breakdown (EleutherAI, "Transformer Math 101")
  • The 6×params×tokens full-fine-tuning compute formula — checked against GPT-3's published training compute; LoRA's ~4×params×tokens variant follows the same forward/backward accounting (Thinking Machines Lab, "LoRA Without Regret," 2025)
  • The activation-memory formula — Korthikanti et al. 2022 (NVIDIA), "Reducing Activation Recomputation in Large Transformer Models"
  • GPU memory size — a spec sheet number

Assumption

  • MFU — the single biggest wildcard; real achieved utilization varies enormously by setup (real large runs on H100-class hardware have reported 30-46%, so the default sits mid-range, not at an extreme)
  • Activation memory implementation multiplier — the formula above is a solid baseline, but flash-attention-style kernels and framework choice still move the real number around
  • Data-parallel scaling efficiency — real multi-GPU sync overhead varies with network quality
  • Blackwell (B200/GB200) peak FLOPS — reported inconsistently across sources; treat as lower-confidence than Hopper
What this still doesn't model

Pipeline parallelism, per-layer sharding floors for very wide models, hybrid within-node/cross-node topology beyond a static warning above ~512 GPUs, checkpoint-saving I/O time, and real interconnect bottlenecks between nodes. All of this pushes a real deployment's numbers around — this tool gets you to a defensible starting order of magnitude, not a purchase order. (ZeRO-3/FSDP sharding — the dominant way large models are actually trained today — is modeled; see Chapter 17.)

PART 3One flow, not three tools
13

Why sizing and cost used to be separate, and why that was friction

Parts 1 and 2 both end the same way: a recommended GPU type and count. But a GPU count on its own doesn't tell you what any of this costs, or whether it's a good investment — that's a completely different question, answered by a completely different model: capex, power, financing, depreciation, pricing, and a set of financial statements and returns.

Originally these lived as separate pages. You'd size your workload, read off a number like "24 GPUs," then go type 24 into a different tool's cluster-size field by hand. That's not a design decision worth defending — it's just friction, and it's exactly the kind of step that goes away when one tool already knows the answer the next one needs.

Think of it like a mortgage calculator that makes you first work out the size of house you can afford somewhere else, on paper, before it'll let you type that number in. The size question and the affordability question are genuinely different — but there's no reason they need to live in different rooms.
One funnel, not three tools Workload Inference or Training Sizing model + use case GPU type + count e.g. 44 × H100 Cost & Returns every field along the way stays editable, all the way to the end
What used to be three separate pages, handed off by hand.
14

What the planner actually carries forward

The planner opens with one question — what are you sizing for — with three doors: Inference, Training, or "I already know my GPU count." The first two lead into Parts 1 and 2 above, unchanged. Whichever you pick, the recommended (or overridden) GPU count and GPU type carry straight into the cost model as its starting point:

What gets carried from sizing into the cost model
From sizingBecomes, in the cost model
Recommended GPU type (e.g. H100)GPU / system type — pre-selects the matching capex, power draw, and VRAM figures
Recommended total GPUs (e.g. 44)Cluster size — carried across exactly, no rounding. A small inference deployment that only needs 4 GPUs stays at 4, not padded up to a full node.
Nothing becomes uneditable

Every field the sizing step fills in — GPU type, capex per GPU, cluster size, all of it — is the exact same input you could type into by hand. A "sized from" note stays visible at the top so you always know where the starting numbers came from, with a link back to adjust the sizing itself. Change your mind about the workload entirely, and "Start over" resets the cost model back to its own defaults rather than leaving stale numbers behind.

15

One more fix this made necessary

Building one flow surfaced something two separate tools had let slide: the sizing tools and the cost model each kept their own table of GPU specs, built in separate sessions, under different names for the same hardware.

The cost model's table knew a GPU's price and power draw. The sizing tools' table knew its memory size, memory bandwidth, and compute speed. Neither table knew the other's numbers, and "NVIDIA H100 SXM" in one wasn't even spelled the same as "NVIDIA H100 (HGX, 8-GPU)" in the other. Harmless as long as the tools never talked to each other — but the moment sizing hands a GPU choice to the cost model, both tables need to agree it's the same part. They're now one shared table, and the on-disk B200 VRAM figure was corrected from 180GB to its actual 192GB spec in the process — a small inconsistency that only became visible once there was one source of truth to check it against.

Two tables become one COST MODEL'S TABLE capex, watts "NVIDIA H100 (HGX, 8-GPU)" SIZING TOOLS' TABLE VRAM, bandwidth, FLOPs "NVIDIA H100 SXM" one shared GPU_PRESETS table
Same part, once spelled two different ways. Now spelled once.
16

A more precise power number, if you want one

The cost model's "IT power draw per GPU" field is a single flat number — by default, a reasonable all-in guess for the selected GPU. But real power draw isn't flat: a GPU sitting near-idle pulls a fraction of what it pulls at full load, and the rest of the server (CPU, NVSwitch fabric, memory, networking, cooling, PSU losses) has its own idle-to-max range on top of that.

Next to that field, a Simple / Detailed toggle lets you swap the flat guess for a number actually derived from your scenario: pick a server chassis, and the tool computes GPU chip power at whatever utilization you've set in Section 5 (Revenue & Pricing) using the same idle-to-TDP curve as the standalone efficiency calculator, adds that chassis's per-GPU share of everything else, and writes the total into the field automatically — recalculating live if you change the utilization assumption. Switch back to Simple at any point and the field becomes a plain editable number again.

Why utilization matters here specifically

The difference between a GPU idling and a GPU at full tilt can be 2-3x in power draw, and that draw is what your colocation or electricity bill is actually metered on. A flat "nameplate-ish" wattage number either overstates your power cost (if you're not running near full load) or understates it (if you are) — Detailed mode ties the two together instead of leaving them as separate guesses.

17

A smarter way to add GPUs: ZeRO-3 / FSDP sharding

Part 2 (Chapters 9-11) sized training GPUs a specific way: find how many GPUs it takes to fit one full copy of the model's training state (call it the "fitting group"), then, if you need more raw compute to hit your deadline, clone that entire group as many times as needed. That's how training was mostly done a few years ago. It's not how it's actually done today.

Real training jobs almost universally use ZeRO-3 or FSDP (DeepSpeed's and PyTorch's names for the same idea) — instead of every GPU holding a full copy of the training state, the state gets split up across every GPU in the job, and each GPU reconstructs the slice it needs on the fly, then discards it. There's no "fitting group" to clone whole. Every GPU you add both shrinks everyone's memory burden and adds compute, continuously, one GPU at a time.

The old way is like a moving crew that only works in fixed teams of exactly 6 — need more speed, hire another whole team of 6, even if you only needed 1 more pair of hands. Sharding is hiring movers one at a time; the load is already split however many ways make sense for however many people showed up.

The planner's training screen now has a ZeRO-3/FSDP vs. Legacy toggle. ZeRO-3/FSDP is the default; Legacy keeps the old fixed-group math around for comparison, or for the rare case you're deliberately planning to NOT shard.

Legacy: clone the whole group ZeRO-3/FSDP: add one at a time 3 GPUteam 3 GPUteam 3 GPUteam 3 GPUteam (2 unused) 4 whole clones = 12 GPUs for a 10-GPU need ONE SHARDED POOL exactly 10 GPUs, no rounding up
Same compute target, 10 GPUs. Legacy pays for 2 it doesn't need.
18

Worked: the same job, both ways

Same scenario as Chapter 9 — Llama 3.1 70B, full fine-tuning, 2-billion-token dataset, 14-day target:

Full fine-tuning — Legacy vs. ZeRO-3/FSDP
MetricLegacyZeRO-3/FSDP
GPUs required to fit1821
GPUs required for compute—2
Total GPUs recommended1821
Counterintuitive, but correct

Here ZeRO-3/FSDP needs more GPUs to fit than Legacy's number, not fewer — worth sitting with, since it's the opposite of what "sharding helps" usually implies. The reason: activation memory is never sharded, by either ZeRO-3 or FSDP — confirmed by both papers this tool cites. Legacy's formula divides everything (training state and activations both) by the fitting-group size, which quietly assumes activation memory shrinks as you add GPUs to the group. It doesn't, in reality, if your per-GPU batch size stays fixed. ZeRO-3/FSDP's formula is the more honest one: each GPU still needs to hold its own full activation memory no matter how many GPUs are in the job, so that fixed cost eats into the shrinking headroom as the training state gets divided more ways — and here, it pushes the answer from 18 to 21.

Now LoRA, same model, tightened to a 2-day deadline (Chapter 11's tighter-deadline example) — this is where the "no cloning a whole group" difference actually shows up:

LoRA, 2-day deadline — Legacy vs. ZeRO-3/FSDP
MetricLegacyZeRO-3/FSDP
GPUs required to fit33
GPUs required for compute—10
Data-parallel replicas / clone count4—
Total GPUs recommended1210
The actual, practical benefit

Both agree the fitting group is 3 GPUs. But Legacy can only add compute by cloning that entire 3-GPU group — needing 10 GPUs' worth of compute forces it to round up to 4 whole clones, 12 GPUs, paying for 2 GPUs it doesn't need. ZeRO-3/FSDP just adds GPUs to the one sharded job until the compute target is met: 10, no rounding up to a group multiple. This is the real benefit of sharding — not really that it always needs fewer GPUs to fit (the table above shows it can need slightly more), but that it never wastes GPUs rounding up to a clone boundary once you're past the fitting question and into the compute one.

PART 4What's new — additions since the original three parts
19

Mixture-of-Experts: the slice that's actually touched

A handful of models in the catalog — Mixtral 8x7B, DeepSeek-V3, Qwen3-235B-A22B — aren't built like the rest. A normal ("dense") model runs every one of its parameters on every single token. A Mixture-of-Experts (MoE) model instead keeps many separate specialist sub-networks — "experts" — and a small router decides, token by token, which handful of experts actually get used. DeepSeek-V3 has 671B parameters total, but only about 37B of them do any work on a given token.

That split matters because memory and compute answer to two different numbers here. Every expert's weights have to sit in GPU memory the whole time — you never know in advance which ones the next token will need, so nothing can be left out. That's total parameters, and it's what the memory-fit math in Chapter 3/Chapter 9 uses, unchanged. But the actual arithmetic work — and therefore speed — only touches whichever experts got routed to. That's active parameters, a much smaller number, and it's what should drive throughput and training-compute math instead.

Same eight experts, two different questions Memory holds all of them · a token's compute only visits a few MEMORY: ALL RESIDENT E1 E2 E3 E4 E5 E6 E7 E8 weights = ALL 8 experts COMPUTE: FEW VISITED tok E1 E2 E3 E4 E5 E6 this token's math = 2 of 8 experts (highlighted)
DeepSeek-V3: 671B total parameters resident in memory, ~37B active on any given token — an ~18× gap between the two numbers.
Total vs. active parameters — stored, vs. touched-this-token
Total parameters is every expert's weights added up — the number that determines whether the model fits in memory at all. Active parameters is only the experts a given token actually routes through — the number that determines how much arithmetic (and therefore GPU time) that token costs. For a dense model the two numbers are identical; for an MoE model, active is a small fraction of total.
MoE compute basis — a toggle, shown only for MoE models
Defaults to Active experts, matching how these models actually run in production serving and how their own creators report training compute (DeepSeek-V3, Mixtral, and Switch Transformer all publish compute figures this way). Switch it to Full / all experts to see a deliberately conservative number instead — useful as a worst-case sanity check, not a recommendation.
A bug this made necessary to fix

An earlier version of the training-sizing math used total parameters for the training-compute calculation too — not just the memory one. For DeepSeek-V3 that overstated the required training compute by roughly 18×, and for Mixtral 8x7B by roughly 3.6× (verified by hand against both models, active vs. total). Fixed now: compute math uses active parameters by default, memory math still (correctly) uses total. If you ever see a wildly larger-than-expected GPU count for an MoE model elsewhere, this is the class of bug to suspect.

20

Reasoning models need a bigger number

Some models — DeepSeek-R1 and o1-style "thinking" models — don't go straight from your question to an answer. They generate a long, hidden chain-of-thought first, working through the problem step by step, and only then write the visible reply you actually see.

From the GPU's point of view, that hidden reasoning is not free. It's generated exactly the same way as any visible output token — one at a time, autoregressively — and it costs exactly the same compute, memory bandwidth, and KV cache space. A tool that only counts the visible answer is quietly ignoring most of what the GPU actually had to do.

It's the difference between timing someone's spoken answer and timing everything they actually did to get there, including the scratch paper. The scratch paper still took time and a desk to work on — it just isn't what you see in the transcript.
Llama 3.1 70B, same use case, reasoning tokens on vs. off
SettingTotal output tokensTarget throughputGPUs recommended
Reasoning tokens = 0 (plain chat)50020,000 tok/s44
Reasoning tokens = 3,0003,500140,000 tok/s287
The field to use

Both inference tools now have a "Reasoning tokens (hidden, before answer)" field, separate from the visible-answer token count, defaulting to 0. It feeds the same context-length, KV-cache, and throughput math as the visible tokens — nothing new to learn, just an honest input. Leave it at 0 for a plain chat/completion model; for a reasoning model, a rough starting range is 1,000–8,000+ tokens depending on task difficulty, tightened up with your own measurements once you have them.

21

Not just NVIDIA: AMD Instinct and Blackwell Ultra

The GPU list covers three generations from each vendor. NVIDIA: H100 and H200 (Hopper), B200 (180GB) and GB200 NVL72, and the Blackwell Ultra parts B300 and GB300 NVL72 (288GB each). AMD: MI300X (192GB, 5.3TB/s), MI325X (256GB, 6TB/s) and MI355X (288GB, 8TB/s, with native FP4).

Two details matter for comparing them fairly. First, power is "all-in" for every GPU: the whole server's draw (CPUs, switches, network cards, fans, power-supply losses) divided by the GPUs in it — not just the chip's own rating. Earlier versions used the chip-only figure for AMD, which made AMD look about 40% cheaper to run than it is. Second, the NVL72 racks join 72 GPUs into one fast domain, so a large mixture-of-experts model can spread across up to 72 GPUs as a single replica; on 8-GPU servers anything bigger than 8 has to be split across nodes with pipeline parallelism, which costs speed (Chapter 25).

Some numbers here are real estimates, not specs

Memory size, bandwidth, and FLOPS come from vendor datasheets — treat those like any other row in Chapter 6's "verified" column. Capex (purchase price) is different: neither vendor publishes list pricing for these parts, so the dollar figures are estimates from market reports and rental-price ratios. Blackwell, Blackwell Ultra and all AMD parts are flagged lower confidence in the tool itself, on both the inference and the cost screens — replace them with your own quote before they drive a real purchasing decision.

22

Trust, but verify: calibrating from a real deployment

Chapter 6 named serving efficiency as the single biggest wildcard in the whole inference model — a reasonable default, but still a guess standing in for something only your own serving stack can actually tell you. If you have a real deployment running, you can replace that guess with your own data.

The calibration section (bottom of the inference sizing inputs) asks for two numbers from one full replica — the same GPU group the tool sized for you — under realistic load: your measured, sustained tokens/sec, and the batch size (concurrent requests) it was running at. The batch matters: 2,000 tok/s at a batch of 8 and at a batch of 256 mean very different things about how efficient your stack is. From those two numbers the tool works backward to the serving efficiency that would have produced them, and sets the slider to it.

It also tells you when a measurement doesn't make sense: if your tokens/sec is above what the hardware could theoretically do at that batch, something's off with the measurement (or the setup doesn't match the inputs), and nothing is applied.

What gets remembered — and what doesn't leave your browser
A calibration is saved locally (in this browser only, never sent anywhere — same "nothing leaves your machine" posture as the encryption on these pages) so it survives a reload. It's offered back to you next time as a banner with Reapply / Discard buttons — it never silently overrides the visible slider on its own. Calibrations saved by the older version of the tool (which tuned a "batching multiplier") can't be converted, because they didn't record the batch size; the banner asks you to re-derive with your current inputs instead.
Estimated cost per 1M tokens — planner-only
The unified planner's inference screen also shows a rough $ per million tokens figure — the hardware's cost recovered over its useful life at your cost of capital (WACC, net of salvage — the same way the full model does it) plus power, divided by the throughput the sizing targets. In colocation mode the power part uses your colocation rate, not utility electricity. It's deliberately simplified (no maintenance, insurance, tax, or utilization ramp — that's what the full Cost & Returns step is for), meant to answer "am I in the right ballpark" against published per-token pricing, not to replace the detailed model.
PART 5Under the hood — how the 2026 models change the math
23

Not all KV caches grow the same way

Chapter 1 described the KV cache as each conversation's short-term memory. In the classic design every layer stores a key and a value for every token, so the memory grows in a straight line with conversation length. Most models released since 2025 don't work that way any more — and treating them as if they did overstates their memory by 2× to 50× at long context.

Full attention — GQA / MHA (Llama 3, Qwen2.5, Mixtral)
Every layer remembers every token. Grouped-query attention (GQA) shares each key/value across several query heads, which shrinks the cache, but it still grows in a straight line with context.
Latent attention — MLA (DeepSeek V3, Kimi K2, Mistral Large 3, GLM-5)
Instead of full keys and values, each layer stores one small compressed vector per token (the "latent," typically 512 numbers plus a 64-number position part). Still linear, but about 5× smaller than a comparable GQA model.
Sliding window — gpt-oss, Gemma 4, Llama 4
Most layers only look back a fixed window (say, the last 128 or 4,096 tokens); a few "global" layers see everything. Past the window, only the global layers keep growing — so memory grows much more slowly than context.
Linear / Mamba hybrids — Qwen3.5/3.6, Nemotron 3
Most layers keep a fixed-size running summary instead of a per-token cache; only every fourth layer or so is real attention. The summary costs the same at 1k tokens as at 1M.
Compressed attention — DeepSeek V4
Groups of tokens are compressed into single entries, plus a small sliding window for recent tokens. A brand-new design, so the tool marks it lower confidence.
KV cache for one conversation, BF16 — the tool's own numbers
ModelAttention type1k tokens32k tokens131k tokens
Llama 3.1 70BFull (GQA)0.33 GB10.7 GB43.0 GB
DeepSeek-V3Latent (MLA)0.07 GB2.3 GB9.2 GB
Gemma 4 31BSliding window0.90 GB3.5 GB11.6 GB
gpt-oss-120bSliding window0.04 GB1.2 GB4.8 GB
Qwen3.5-397B-A17BLinear hybrid0.22 GB1.2 GB4.2 GB
DeepSeek-V4-FlashCompressed0.01 GB0.20 GB0.78 GB
WHAT EACH LAYER KEEPS, FOR A 12-TOKEN CONVERSATION Full (GQA) every token, full size Sliding window only the last 4 tokens Latent (MLA) every token, compressed
Filled boxes are kept in memory. Sliding-window layers forget everything outside the window; latent layers keep everything, but each entry is a fraction of the size.
What this means for sizing

For long-context work (RAG, coding agents, document analysis), the attention type can matter more than the parameter count. A 397B hybrid model can need less memory per conversation at 131k tokens than a 70B full-attention model at 32k.

24

FP4, and why the KV cache has its own precision

Blackwell and MI355X GPUs run 4-bit math natively, and several 2026 models ship in a 4-bit format. Three 4-bit formats are in use, and none of them is exactly half a byte:

Bytes per parameter, including the shared scales
FormatHow it's storedBytes / param
BF162-byte floating point — the training default2.0
FP81-byte floating point1.0
NVFP44-bit values, plus one 8-bit scale per 16 values (NVIDIA)0.5625
MXFP44-bit values, plus one 8-bit scale per 32 values (OCP standard; gpt-oss ships this way)0.53
INT44-bit integers with group scales (GPTQ/AWQ style)~0.53

"Native" means "whatever format the model's publisher ships." For a model released in FP8 or MXFP4, that's the honest starting point — re-quantizing it costs effort and accuracy for no memory gain.

On GPUs without FP4 hardware (H100, H200, MI300X, MI325X), the tool still lets you store weights in 4 bits — that saves memory and bandwidth — but runs the math at BF16 speed, and says so in a warning.

KV-cache precision is a separate input. The weights and the conversation memory don't have to share a format: many deployments run FP4 weights with an FP8 KV cache, or BF16 weights with an FP8 cache. "Auto" picks FP8 when you choose FP8-or-smaller weights and BF16 otherwise. For a model served exactly as shipped it follows the serving stack's default — BF16, except where the model's reference setup uses FP8 (DeepSeek V3.2 and V4). Latent-attention caches don't quite halve in FP8: the position part stays in BF16 and scale factors are added, so DeepSeek's 1,152 bytes per token per layer become 656, not 576. Since the KV cache is often the larger half of the memory bill (Chapter 3), this one setting can nearly double how many conversations fit.

25

The roofline: how batch size, speed and team shape interact

Earlier versions used one fixed "batching multiplier." The tool now models a decoding step directly, the way GPU engineers do — with a roofline.

Each step has two costs. Memory time: read every weight once (for MoE models, only the experts the batch actually touches — Chapter 19), plus every conversation's KV cache. Compute time: about 2 math operations per active parameter, per person in the batch. The step takes whichever is longer, divided by the serving efficiency. At small batches memory dominates, so adding people is almost free; at large batches compute takes over and every extra person slows everyone down.

The tool then searches for B*: the largest batch whose per-person speed still meets your response-time target, capped by how many KV caches fit in memory. Replicas = concurrent sessions ÷ B*, rounded up. If even one person can't be served fast enough, the target is not reachable on that hardware, and the planner won't hand that number to the cost model unless you override it.

Team shape — TP, wide expert parallel, PP
Tensor-parallel teams are 1, 2, 4 or 8 GPUs, and must divide the model's attention heads evenly. On NVL72 racks, a mixture-of-experts model can use a wide team of 16–72 GPUs. Anywhere else, a model that needs more than 8 GPUs is split into pipeline stages across servers; each extra stage costs speed (pipeline efficiency, default 90% per hop) and the result is labelled multi-node. For MoE models the tool also tries data-parallel attention: every GPU keeps a full copy of the shared (non-expert) weights, but only its own conversations' KV cache — it wins when memory for conversations, not weights, is the limit.
Per-step overhead — the speed floor
Every generated token is one step for the whole batch. On top of reading memory and doing math, each step pays a small fixed cost — about 0.3 ms for scheduling and launching work, plus two quick sync-ups per layer whenever a layer is split across GPUs. It's invisible for big dense models but caps how fast small or very sparse models can go.
Time to first token — TTFT
Before writing anything, the model reads the whole prompt in one pass (the "prefill"). The tool estimates this as the longer of the prompt's math and one full read of the weights, and shows it next to the per-replica throughput. Prefill compute also counts against the GPU's busy time, so long prompts reduce how many people a replica can serve. The tool assumes each session sends its next request as soon as the last one finishes (or once per target time, if that's longer), so a loose target never inflates the tokens-per-GPU figure.
Sanity check — against published benchmarks
With FP8, 4,000 sessions and a relaxed 60-second target, Llama 70B on H100 comes out at about 2,800 tok/s per GPU — inside the range of public MLPerf inference results for that setup. Mixture-of-experts models are less well anchored: gpt-oss-120b on H100 comes out about 10% above public figures and DeepSeek-V3 on GB200 about 20% below a heavily tuned large-scale deployment, so calibrate before committing to an MoE fleet. The test suite checks this on every change.
26

Where the model numbers come from — and adding your own

Every model in the catalog was entered from its official config.json on Hugging Face. The hint under the model picker shows the repository it came from (as a normal link you can click) and the date it was last checked, so you can verify any number yourself.

The tool never fetches anything from Hugging Face or anywhere else — the catalog is built into the page, consistent with the "nothing leaves your machine" posture of the whole site. The trade-off is that new models appear when the catalog is updated, not the day they're released.

For anything missing, pick Custom model (last in the list) and type the architecture in from the model's config.json: parameter counts, layers, heads, head size, context length, and the attention type — which reveals only the fields that type needs (for example, the latent rank for MLA or the window size for sliding attention). Multimodal models keep the language part under text_config. The form checks the numbers make sense (heads divide evenly, active parameters don't exceed total) before sizing anything, and remembers your entry in this browser only.

Lower-confidence entries

Brand-new architectures (DeepSeek V4's compressed attention) and recently launched GPUs are flagged lower confidence. The structure is right, but there's no long track record of real deployments to check the numbers against yet. Treat their outputs as a first estimate and calibrate (Chapter 22) once you have a real measurement.