How the planner actually works
Every number the planner shows you, explained in plain language — so you can trust it, argue with it, and catch it when it's wrong. Part one is inference. Part two is training and fine-tuning. Part three is how those two connect to the cost model, and why they're one tool now instead of three. Part four covers what's been added most recently.
Two separate questions
Everything the tool does comes down to answering two independent questions about a model you want to run:
1. Does it fit? The model itself, plus every conversation currently happening, has to physically sit in the GPU's memory at the same time. If it doesn't fit, nothing else matters — the system can't even start.
2. Is it fast enough? Once it fits, can it produce answers quickly enough for however many people are using it, given how fast you've promised they'll get a reply?
The tool answers both questions separately, then takes whichever one needs more GPUs — because you have to clear both bars, not just one.
Vocabulary, before anything else
Ten words carry the whole tool. Once these click, the rest is arithmetic.
Precision has a wrinkle worth understanding on its own: how a model gets squeezed down to a lower precision affects how much accuracy survives the trip — but not the memory math this tool does.
Naive rounding breaks down on large models because of outliers — roughly 0.1% of a model's internal values run up to 20× larger than everything around them, and a rounding grid built wide enough to fit those few extreme values wastes almost all its resolution on the other 99.9%. Five well-known techniques each solve this outlier problem at a different point in the pipeline:
All five techniques above produce the same artifact: a model stored at a given bytes-per-parameter, whichever one made it there. They differ in how much accuracy survives the trip — not in how much memory the result takes up. That's the whole reason this tool's precision input is just a format (BF16 / FP8 / NVFP4 / MXFP4 / INT4) with no method picker alongside it: memory footprint is method-agnostic, accuracy is not, and sizing only needs the former. (Source: a comprehensive survey of these methods at arXiv:2411.02530.)
Question 1, worked: does it fit?
Using the tool's own default scenario — Llama 3.1 70B, BF16, 200 concurrent chats, 500 tokens in / 500 tokens out, a reply inside 10 seconds, on H100s — here's what each row of the "Memory Constraint" table actually means:
| Number | What it means | Value |
|---|---|---|
| Model weights | 70.6 billion numbers × 2 bytes each (BF16) = the model itself, installed. | 141.2 GB |
| KV cache, one conversation | The short-term memory for a single 1,000-token exchange (500 in + 500 out). | 0.33 GB |
| KV cache, all conversations | That 0.33 GB, times 200 people chatting simultaneously. | 65.5 GB |
| Safety margin | Buffer for the serving software's own runtime, ~10% of the above. | 20.7 GB |
| Total memory needed | Everything above, added up — what has to fit somewhere. | 227.4 GB |
| Usable space per GPU | An H100's 80GB, minus ~10% the system always reserves for itself. | 72 GB |
| → Smallest team that fits | Teams come in sizes of 1, 2, 4 or 8 GPUs (the model's attention heads have to split evenly). 4 H100s hold the weights with 33 GB per GPU left over for conversations — room for about 368 of them. | 4 |
A 70-billion-parameter model fits on a team of 4 H100s with room to spare for its conversations — but fitting is only the first question, and here it isn't the one that decides the answer.
Question 2, worked: is it fast enough?
A 500-token reply inside 10 seconds means every person needs at least 50 tokens per second streamed back to them. Each decoding step has to read every weight from memory, so a team's speed for a single person is capped by how fast it can read the model:
| Number | What it means | Value |
|---|---|---|
| Each person needs | 500 tokens ÷ 10 seconds. | 50 tok/s |
| 4-GPU team, one conversation | The fastest a 4-GPU team can stream to a single person, at 50% serving efficiency. Below 50 — so the team that fits is too slow, no matter how few people use it. | 38 tok/s |
| 8-GPU team, one conversation | Twice the GPUs reading the model in parallel. Not quite twice as fast — every step also pays a fixed cost for the 8 GPUs to sync up — but there's headroom above 50. | 71 tok/s |
| Largest batch that stays fast enough | The tool grows the batch until each person's speed drops to just above 50 tok/s. | 93 people |
| The 8-GPU team, serving that batch | 93 people × ~50 tok/s each — about 584 tok/s per GPU. Still limited by memory bandwidth, not compute. | 4,672 tok/s |
| What 200 people actually need | 200 people × 50 tok/s. | 10,000 tok/s |
| → Teams needed | 200 people ÷ 93 per team, rounded up. | 3 |
4 GPUs can hold the model, but only 8 can stream it fast enough for one person — and 200 people need 3 of those 8-GPU teams, so the final answer is 24 GPUs.
Why "the bigger number," not "add them up"
Question 1 said "a team of 4 fits." Question 2 said "a team of 8 is needed for speed, and 3 of them." The tool takes the team size that satisfies both — 8 — and then asks how many of those teams the traffic needs: 8 × 3 = 24. Fitting and speed aren't stacked costs; the speed answer is "how many teams," and each team is already at least the size that fits.
If your use case had been 10 people instead of 200, one 8-GPU team would have been enough — the answer would stay at 8 GPUs, because one team was always going to serve them all. And if you'd asked for a 5-second reply instead of 10, each person would need 100 tok/s — more than even a single 8-GPU H100 team can deliver to one person. The tool flags that as not reachable rather than pretending more GPUs will fix it: you need faster GPUs, lower precision, or a looser target.
Lower precision moves both questions at once. Switch the same scenario to FP8 and the weights halve, so each step reads half as much: a single 8-GPU team now streams 128 tok/s to one person and can serve all 200 at once. The answer drops to 8 GPUs — about three times the tokens per GPU.
What's a fact, what's a guess
Not every number on the page carries the same weight. Some come from real, published hardware and model specifications. Others are the tool's best estimate of something genuinely uncertain — and you should treat those as a starting point to test, not a promise.
Verified
- Model size (params) — publicly documented per model
- Model's internal shape (layers, memory-per-token) — from the model's own published config
- GPU memory size (VRAM) — a spec sheet number
- GPU memory speed (bandwidth) — a spec sheet number
- Bytes-per-number at each precision — a fixed fact of BF16/FP8/FP4, including the 4-bit formats' shared scales
- Memory footprint at a given precision, regardless of quantization method (RTN/GPTQ/AWQ/LLM.int8()/QAT) — see Chapter 2
Assumption
- Serving efficiency — how close your software gets to the hardware's theoretical speed (default 50%); the biggest wildcard by far, so calibrate it from your own stack
- Pipeline efficiency — the cost of splitting a model across nodes (default 90% per hop)
- Tensor-parallel efficiency — real-world team coordination is never perfectly linear
- Usable VRAM % — how much the runtime quietly reserves for itself
- Framework overhead % — a rough buffer, not a measured number
- Your own traffic pattern — real usage is bursty, not a smooth constant
Trust the left column as-is. Treat every number in the right column as a slider to stress-test — run the tool once optimistic, once conservative, and see how much the answer actually moves. If it swings wildly, that's the assumption worth validating with a real benchmark before you buy anything.
A different kind of question
Inference answers questions live, one exchange at a time, forever. Training is the opposite shape: one long, continuous run that has to get through a fixed amount of material once, then stop. The two questions that mattered for inference still apply, but they mean something different:
1. Does one copy fit? Before training can even start, the model plus everything the training process itself needs to remember has to sit in memory together — and that "everything else" turns out to be much bigger than the model alone.
2. How long will it take, and how many GPUs does that mean? Given the total amount of work the job requires, and a deadline you're willing to accept, how many GPUs do you need running together to get there in time?
Vocabulary, part two
A few new words, and one you already know wearing a different hat.
Question 1, worked: does one copy fit?
Same model as before — Llama 3.1 70B — but now full fine-tuning on a 2-billion-token dataset:
| Number | What it means | Value |
|---|---|---|
| Weights + gradients + optimizer state | 70 billion parameters × ~16 bytes each — the model, its correction signals, and the optimizer's own memory, all at once. | 1.12 TB |
| Activation memory | Scratch-work for a batch of 2 sequences at 4,096 tokens each, with full activation recomputation (checkpointing) enabled. | 10.7 GB |
| Safety margin | Buffer for the training framework's own runtime, ~10% of the above. | 113.1 GB |
| Total memory needed | Everything above, added up. | 1.24 TB |
| Usable space per GPU | An H100's 80GB, minus the runtime's own reserve. | 72 GB |
| → GPUs needed to fit one copy | 1.24 TB ÷ 72 GB, rounded up. | 18 |
Fully fine-tuning a 70B model needs 18 GPUs working together just to have somewhere to put the model, its corrections, and the optimizer's memory — before asking how long the job takes.
Why LoRA changes the answer so much
Same model, same 2-billion-token job — but train only the small LoRA slice (~1% of parameters) instead of the whole thing:
| Metric | Full fine-tuning | LoRA |
|---|---|---|
| Memory needed | 1.24 TB | 178 GB |
| GPUs to fit one copy | 18 | 3 |
| Total training compute | 8.4 × 10²⁰ FLOPs | 5.6 × 10²⁰ FLOPs |
| Total GPUs recommended | 18 | 3 |
LoRA's compute row is lower, but not proportionally lower — about ⅔ of full fine-tuning's, not ~1% even though only ~1% of parameters are being trained. Every forward pass still touches all 70B parameters, frozen or not. In the backward pass, every parameter — frozen or trainable — still needs its input-gradient computed, since that's the signal relayed further back through the network to eventually reach the trainable adapters. What frozen parameters skip is their own weight-gradient — the update instruction, which only a trainable parameter needs. Skipping that halves the backward-pass cost for the ~99% of parameters that are frozen, which is where the savings come from. LoRA is mostly a memory trick with a real, smaller compute saving on top — memory drops by ~7x here (1.24TB → 178GB) while compute drops by only ~1.5x, which is why the GPU count is dominated by the memory constraint (3, not something compute-derived) at this dataset size.
Question 2, worked: how long, and how many
Back to full fine-tuning's 18 GPUs. How long does the actual job take, and does it need more GPUs than the fitting question alone required?
| Number | What it means | Value |
|---|---|---|
| Total training compute | 6 × 70 billion parameters × 2 billion tokens — the standard estimate for a full forward-and-backward pass over the whole dataset. | 8.4 × 10²⁰ FLOPs |
| Per-GPU achievable speed | An H100's spec-sheet peak, discounted to the 40% of it real training actually sustains. | 396 TFLOP/s |
| Target training time | The deadline: what you told the tool you're willing to wait. | 14 days |
| → Extra copies needed to hit that deadline | The 18 GPUs already required just to fit turn out to be so much combined power that they clear the 14-day target easily — no extra copies needed. | 1 |
| Actual time at 18 GPUs | Far under budget — a side effect of needing that many GPUs just to fit, not because speed demanded it. | 1.5 days |
That's the opposite of Part 1's example, where speed forced extra GPUs beyond what fitting alone required. Here it's flipped — full fine-tuning at this scale is so memory-hungry that fitting alone gives you far more speed than you asked for. To see the other direction, tighten LoRA's deadline from 14 days to 2 days, keeping everything else the same:
| Metric | 14-day deadline | 2-day deadline |
|---|---|---|
| GPUs to fit one copy | 3 | 3 |
| Data-parallel copies needed | 1 | 4 |
| Total GPUs | 3 | 12 |
| Actual time achieved | 6.1 days | 1.5 days |
Same 3-GPU team, just run 4 identical copies side by side, each working through a different quarter of the dataset at the same time, then combining what they learned. That's data parallelism — the training equivalent of Part 1's "open more replicas" move.
What's a fact, what's a guess — training edition
Same discipline as Part 1. Some numbers are load-bearing facts; others are the tool's best estimate of something genuinely contested.
Verified
- Model size and shape — published per model
- The ~16 bytes/parameter mixed-precision Adam figure — a standard, well-documented breakdown (EleutherAI, "Transformer Math 101")
- The 6×params×tokens full-fine-tuning compute formula — checked against GPT-3's published training compute; LoRA's ~4×params×tokens variant follows the same forward/backward accounting (Thinking Machines Lab, "LoRA Without Regret," 2025)
- The activation-memory formula — Korthikanti et al. 2022 (NVIDIA), "Reducing Activation Recomputation in Large Transformer Models"
- GPU memory size — a spec sheet number
Assumption
- MFU — the single biggest wildcard; real achieved utilization varies enormously by setup (real large runs on H100-class hardware have reported 30-46%, so the default sits mid-range, not at an extreme)
- Activation memory implementation multiplier — the formula above is a solid baseline, but flash-attention-style kernels and framework choice still move the real number around
- Data-parallel scaling efficiency — real multi-GPU sync overhead varies with network quality
- Blackwell (B200/GB200) peak FLOPS — reported inconsistently across sources; treat as lower-confidence than Hopper
Pipeline parallelism, per-layer sharding floors for very wide models, hybrid within-node/cross-node topology beyond a static warning above ~512 GPUs, checkpoint-saving I/O time, and real interconnect bottlenecks between nodes. All of this pushes a real deployment's numbers around — this tool gets you to a defensible starting order of magnitude, not a purchase order. (ZeRO-3/FSDP sharding — the dominant way large models are actually trained today — is modeled; see Chapter 17.)
Why sizing and cost used to be separate, and why that was friction
Parts 1 and 2 both end the same way: a recommended GPU type and count. But a GPU count on its own doesn't tell you what any of this costs, or whether it's a good investment — that's a completely different question, answered by a completely different model: capex, power, financing, depreciation, pricing, and a set of financial statements and returns.
Originally these lived as separate pages. You'd size your workload, read off a number like "24 GPUs," then go type 24 into a different tool's cluster-size field by hand. That's not a design decision worth defending — it's just friction, and it's exactly the kind of step that goes away when one tool already knows the answer the next one needs.
What the planner actually carries forward
The planner opens with one question — what are you sizing for — with three doors: Inference, Training, or "I already know my GPU count." The first two lead into Parts 1 and 2 above, unchanged. Whichever you pick, the recommended (or overridden) GPU count and GPU type carry straight into the cost model as its starting point:
| From sizing | Becomes, in the cost model |
|---|---|
| Recommended GPU type (e.g. H100) | GPU / system type — pre-selects the matching capex, power draw, and VRAM figures |
| Recommended total GPUs (e.g. 44) | Cluster size — carried across exactly, no rounding. A small inference deployment that only needs 4 GPUs stays at 4, not padded up to a full node. |
Every field the sizing step fills in — GPU type, capex per GPU, cluster size, all of it — is the exact same input you could type into by hand. A "sized from" note stays visible at the top so you always know where the starting numbers came from, with a link back to adjust the sizing itself. Change your mind about the workload entirely, and "Start over" resets the cost model back to its own defaults rather than leaving stale numbers behind.
One more fix this made necessary
Building one flow surfaced something two separate tools had let slide: the sizing tools and the cost model each kept their own table of GPU specs, built in separate sessions, under different names for the same hardware.
The cost model's table knew a GPU's price and power draw. The sizing tools' table knew its memory size, memory bandwidth, and compute speed. Neither table knew the other's numbers, and "NVIDIA H100 SXM" in one wasn't even spelled the same as "NVIDIA H100 (HGX, 8-GPU)" in the other. Harmless as long as the tools never talked to each other — but the moment sizing hands a GPU choice to the cost model, both tables need to agree it's the same part. They're now one shared table, and the on-disk B200 VRAM figure was corrected from 180GB to its actual 192GB spec in the process — a small inconsistency that only became visible once there was one source of truth to check it against.
A more precise power number, if you want one
The cost model's "IT power draw per GPU" field is a single flat number — by default, a reasonable all-in guess for the selected GPU. But real power draw isn't flat: a GPU sitting near-idle pulls a fraction of what it pulls at full load, and the rest of the server (CPU, NVSwitch fabric, memory, networking, cooling, PSU losses) has its own idle-to-max range on top of that.
Next to that field, a Simple / Detailed toggle lets you swap the flat guess for a number actually derived from your scenario: pick a server chassis, and the tool computes GPU chip power at whatever utilization you've set in Section 5 (Revenue & Pricing) using the same idle-to-TDP curve as the standalone efficiency calculator, adds that chassis's per-GPU share of everything else, and writes the total into the field automatically — recalculating live if you change the utilization assumption. Switch back to Simple at any point and the field becomes a plain editable number again.
The difference between a GPU idling and a GPU at full tilt can be 2-3x in power draw, and that draw is what your colocation or electricity bill is actually metered on. A flat "nameplate-ish" wattage number either overstates your power cost (if you're not running near full load) or understates it (if you are) — Detailed mode ties the two together instead of leaving them as separate guesses.
A smarter way to add GPUs: ZeRO-3 / FSDP sharding
Part 2 (Chapters 9-11) sized training GPUs a specific way: find how many GPUs it takes to fit one full copy of the model's training state (call it the "fitting group"), then, if you need more raw compute to hit your deadline, clone that entire group as many times as needed. That's how training was mostly done a few years ago. It's not how it's actually done today.
Real training jobs almost universally use ZeRO-3 or FSDP (DeepSpeed's and PyTorch's names for the same idea) — instead of every GPU holding a full copy of the training state, the state gets split up across every GPU in the job, and each GPU reconstructs the slice it needs on the fly, then discards it. There's no "fitting group" to clone whole. Every GPU you add both shrinks everyone's memory burden and adds compute, continuously, one GPU at a time.
The planner's training screen now has a ZeRO-3/FSDP vs. Legacy toggle. ZeRO-3/FSDP is the default; Legacy keeps the old fixed-group math around for comparison, or for the rare case you're deliberately planning to NOT shard.
Worked: the same job, both ways
Same scenario as Chapter 9 — Llama 3.1 70B, full fine-tuning, 2-billion-token dataset, 14-day target:
| Metric | Legacy | ZeRO-3/FSDP |
|---|---|---|
| GPUs required to fit | 18 | 21 |
| GPUs required for compute | — | 2 |
| Total GPUs recommended | 18 | 21 |
Here ZeRO-3/FSDP needs more GPUs to fit than Legacy's number, not fewer — worth sitting with, since it's the opposite of what "sharding helps" usually implies. The reason: activation memory is never sharded, by either ZeRO-3 or FSDP — confirmed by both papers this tool cites. Legacy's formula divides everything (training state and activations both) by the fitting-group size, which quietly assumes activation memory shrinks as you add GPUs to the group. It doesn't, in reality, if your per-GPU batch size stays fixed. ZeRO-3/FSDP's formula is the more honest one: each GPU still needs to hold its own full activation memory no matter how many GPUs are in the job, so that fixed cost eats into the shrinking headroom as the training state gets divided more ways — and here, it pushes the answer from 18 to 21.
Now LoRA, same model, tightened to a 2-day deadline (Chapter 11's tighter-deadline example) — this is where the "no cloning a whole group" difference actually shows up:
| Metric | Legacy | ZeRO-3/FSDP |
|---|---|---|
| GPUs required to fit | 3 | 3 |
| GPUs required for compute | — | 10 |
| Data-parallel replicas / clone count | 4 | — |
| Total GPUs recommended | 12 | 10 |
Both agree the fitting group is 3 GPUs. But Legacy can only add compute by cloning that entire 3-GPU group — needing 10 GPUs' worth of compute forces it to round up to 4 whole clones, 12 GPUs, paying for 2 GPUs it doesn't need. ZeRO-3/FSDP just adds GPUs to the one sharded job until the compute target is met: 10, no rounding up to a group multiple. This is the real benefit of sharding — not really that it always needs fewer GPUs to fit (the table above shows it can need slightly more), but that it never wastes GPUs rounding up to a clone boundary once you're past the fitting question and into the compute one.
Mixture-of-Experts: the slice that's actually touched
A handful of models in the catalog — Mixtral 8x7B, DeepSeek-V3, Qwen3-235B-A22B — aren't built like the rest. A normal ("dense") model runs every one of its parameters on every single token. A Mixture-of-Experts (MoE) model instead keeps many separate specialist sub-networks — "experts" — and a small router decides, token by token, which handful of experts actually get used. DeepSeek-V3 has 671B parameters total, but only about 37B of them do any work on a given token.
That split matters because memory and compute answer to two different numbers here. Every expert's weights have to sit in GPU memory the whole time — you never know in advance which ones the next token will need, so nothing can be left out. That's total parameters, and it's what the memory-fit math in Chapter 3/Chapter 9 uses, unchanged. But the actual arithmetic work — and therefore speed — only touches whichever experts got routed to. That's active parameters, a much smaller number, and it's what should drive throughput and training-compute math instead.
An earlier version of the training-sizing math used total parameters for the training-compute calculation too — not just the memory one. For DeepSeek-V3 that overstated the required training compute by roughly 18×, and for Mixtral 8x7B by roughly 3.6× (verified by hand against both models, active vs. total). Fixed now: compute math uses active parameters by default, memory math still (correctly) uses total. If you ever see a wildly larger-than-expected GPU count for an MoE model elsewhere, this is the class of bug to suspect.
Reasoning models need a bigger number
Some models — DeepSeek-R1 and o1-style "thinking" models — don't go straight from your question to an answer. They generate a long, hidden chain-of-thought first, working through the problem step by step, and only then write the visible reply you actually see.
From the GPU's point of view, that hidden reasoning is not free. It's generated exactly the same way as any visible output token — one at a time, autoregressively — and it costs exactly the same compute, memory bandwidth, and KV cache space. A tool that only counts the visible answer is quietly ignoring most of what the GPU actually had to do.
| Setting | Total output tokens | Target throughput | GPUs recommended |
|---|---|---|---|
| Reasoning tokens = 0 (plain chat) | 500 | 20,000 tok/s | 44 |
| Reasoning tokens = 3,000 | 3,500 | 140,000 tok/s | 287 |
Both inference tools now have a "Reasoning tokens (hidden, before answer)" field, separate from the visible-answer token count, defaulting to 0. It feeds the same context-length, KV-cache, and throughput math as the visible tokens — nothing new to learn, just an honest input. Leave it at 0 for a plain chat/completion model; for a reasoning model, a rough starting range is 1,000–8,000+ tokens depending on task difficulty, tightened up with your own measurements once you have them.
Not just NVIDIA: AMD Instinct and Blackwell Ultra
The GPU list covers three generations from each vendor. NVIDIA: H100 and H200 (Hopper), B200 (180GB) and GB200 NVL72, and the Blackwell Ultra parts B300 and GB300 NVL72 (288GB each). AMD: MI300X (192GB, 5.3TB/s), MI325X (256GB, 6TB/s) and MI355X (288GB, 8TB/s, with native FP4).
Two details matter for comparing them fairly. First, power is "all-in" for every GPU: the whole server's draw (CPUs, switches, network cards, fans, power-supply losses) divided by the GPUs in it — not just the chip's own rating. Earlier versions used the chip-only figure for AMD, which made AMD look about 40% cheaper to run than it is. Second, the NVL72 racks join 72 GPUs into one fast domain, so a large mixture-of-experts model can spread across up to 72 GPUs as a single replica; on 8-GPU servers anything bigger than 8 has to be split across nodes with pipeline parallelism, which costs speed (Chapter 25).
Memory size, bandwidth, and FLOPS come from vendor datasheets — treat those like any other row in Chapter 6's "verified" column. Capex (purchase price) is different: neither vendor publishes list pricing for these parts, so the dollar figures are estimates from market reports and rental-price ratios. Blackwell, Blackwell Ultra and all AMD parts are flagged lower confidence in the tool itself, on both the inference and the cost screens — replace them with your own quote before they drive a real purchasing decision.
Trust, but verify: calibrating from a real deployment
Chapter 6 named serving efficiency as the single biggest wildcard in the whole inference model — a reasonable default, but still a guess standing in for something only your own serving stack can actually tell you. If you have a real deployment running, you can replace that guess with your own data.
The calibration section (bottom of the inference sizing inputs) asks for two numbers from one full replica — the same GPU group the tool sized for you — under realistic load: your measured, sustained tokens/sec, and the batch size (concurrent requests) it was running at. The batch matters: 2,000 tok/s at a batch of 8 and at a batch of 256 mean very different things about how efficient your stack is. From those two numbers the tool works backward to the serving efficiency that would have produced them, and sets the slider to it.
It also tells you when a measurement doesn't make sense: if your tokens/sec is above what the hardware could theoretically do at that batch, something's off with the measurement (or the setup doesn't match the inputs), and nothing is applied.
Not all KV caches grow the same way
Chapter 1 described the KV cache as each conversation's short-term memory. In the classic design every layer stores a key and a value for every token, so the memory grows in a straight line with conversation length. Most models released since 2025 don't work that way any more — and treating them as if they did overstates their memory by 2× to 50× at long context.
| Model | Attention type | 1k tokens | 32k tokens | 131k tokens |
|---|---|---|---|---|
| Llama 3.1 70B | Full (GQA) | 0.33 GB | 10.7 GB | 43.0 GB |
| DeepSeek-V3 | Latent (MLA) | 0.07 GB | 2.3 GB | 9.2 GB |
| Gemma 4 31B | Sliding window | 0.90 GB | 3.5 GB | 11.6 GB |
| gpt-oss-120b | Sliding window | 0.04 GB | 1.2 GB | 4.8 GB |
| Qwen3.5-397B-A17B | Linear hybrid | 0.22 GB | 1.2 GB | 4.2 GB |
| DeepSeek-V4-Flash | Compressed | 0.01 GB | 0.20 GB | 0.78 GB |
For long-context work (RAG, coding agents, document analysis), the attention type can matter more than the parameter count. A 397B hybrid model can need less memory per conversation at 131k tokens than a 70B full-attention model at 32k.
FP4, and why the KV cache has its own precision
Blackwell and MI355X GPUs run 4-bit math natively, and several 2026 models ship in a 4-bit format. Three 4-bit formats are in use, and none of them is exactly half a byte:
| Format | How it's stored | Bytes / param |
|---|---|---|
| BF16 | 2-byte floating point — the training default | 2.0 |
| FP8 | 1-byte floating point | 1.0 |
| NVFP4 | 4-bit values, plus one 8-bit scale per 16 values (NVIDIA) | 0.5625 |
| MXFP4 | 4-bit values, plus one 8-bit scale per 32 values (OCP standard; gpt-oss ships this way) | 0.53 |
| INT4 | 4-bit integers with group scales (GPTQ/AWQ style) | ~0.53 |
"Native" means "whatever format the model's publisher ships." For a model released in FP8 or MXFP4, that's the honest starting point — re-quantizing it costs effort and accuracy for no memory gain.
On GPUs without FP4 hardware (H100, H200, MI300X, MI325X), the tool still lets you store weights in 4 bits — that saves memory and bandwidth — but runs the math at BF16 speed, and says so in a warning.
KV-cache precision is a separate input. The weights and the conversation memory don't have to share a format: many deployments run FP4 weights with an FP8 KV cache, or BF16 weights with an FP8 cache. "Auto" picks FP8 when you choose FP8-or-smaller weights and BF16 otherwise. For a model served exactly as shipped it follows the serving stack's default — BF16, except where the model's reference setup uses FP8 (DeepSeek V3.2 and V4). Latent-attention caches don't quite halve in FP8: the position part stays in BF16 and scale factors are added, so DeepSeek's 1,152 bytes per token per layer become 656, not 576. Since the KV cache is often the larger half of the memory bill (Chapter 3), this one setting can nearly double how many conversations fit.
The roofline: how batch size, speed and team shape interact
Earlier versions used one fixed "batching multiplier." The tool now models a decoding step directly, the way GPU engineers do — with a roofline.
Each step has two costs. Memory time: read every weight once (for MoE models, only the experts the batch actually touches — Chapter 19), plus every conversation's KV cache. Compute time: about 2 math operations per active parameter, per person in the batch. The step takes whichever is longer, divided by the serving efficiency. At small batches memory dominates, so adding people is almost free; at large batches compute takes over and every extra person slows everyone down.
The tool then searches for B*: the largest batch whose per-person speed still meets your response-time target, capped by how many KV caches fit in memory. Replicas = concurrent sessions ÷ B*, rounded up. If even one person can't be served fast enough, the target is not reachable on that hardware, and the planner won't hand that number to the cost model unless you override it.
Where the model numbers come from — and adding your own
Every model in the catalog was entered from its official config.json on Hugging Face. The hint under the model picker shows the repository it came from (as a normal link you can click) and the date it was last checked, so you can verify any number yourself.
The tool never fetches anything from Hugging Face or anywhere else — the catalog is built into the page, consistent with the "nothing leaves your machine" posture of the whole site. The trade-off is that new models appear when the catalog is updated, not the day they're released.
For anything missing, pick Custom model (last in the list) and type the architecture in from the model's config.json: parameter counts, layers, heads, head size, context length, and the attention type — which reveals only the fields that type needs (for example, the latent rank for MLA or the window size for sliding attention). Multimodal models keep the language part under text_config. The form checks the numbers make sense (heads divide evenly, active parameters don't exceed total) before sizing anything, and remembers your entry in this browser only.
Brand-new architectures (DeepSeek V4's compressed attention) and recently launched GPUs are flagged lower confidence. The structure is right, but there's no long track record of real deployments to check the numbers against yet. Treat their outputs as a first estimate and calibrate (Chapter 22) once you have a real measurement.