GPU Inference Sizing SKETCH

Pick a model + a use case → get a recommended GPU type & count, with the full math shown. Feeds the TCO calculator's cluster size — not yet wired up.
Runs entirely client-side. Sizing is a planning heuristic based on published model architectures — not a capacity guarantee. Override anything.
1. Model
FP4 formats run on FP4 tensor cores on Blackwell and MI355X. On Hopper/MI300 they still save memory but compute at BF16 rate (weight-only quantization).
Serving stacks (vLLM, SGLang, TensorRT-LLM) quantize the KV cache separately from the weights — it is never stored at 4-bit here.
2. Use Case (Interactive Inference)
Visible answer tokens only — see "Reasoning tokens" below for thinking-mode models.
For thinking/reasoning models (e.g. DeepSeek-R1, o1-style) that generate a hidden chain-of-thought before the visible answer. These tokens are generated autoregressively exactly like any other output token — same compute, bandwidth, and KV cache cost — so leaving this at 0 for a reasoning workload will undersize the cluster. Typical reasoning traces run 1,000–8,000+ tokens depending on task difficulty. Leave at 0 for non-reasoning models.
Time to deliver a complete response (prompt processing + all output tokens). It sets the per-user speed every stream must get: (output + reasoning tokens) ÷ (response time − time-to-first-token). Bigger batches raise total throughput but slow each stream, so this caps how many sessions one replica can serve.
3. GPU
4. Advanced / Contested Assumptions
Fraction of nameplate VRAM actually usable after CUDA context, framework runtime, etc.
Discount for NVLink all-reduce traffic when one replica spans several GPUs.
Only used when a model is too big for one NVLink domain and is split into pipeline stages across nodes (bubbles + inter-node hops).
The most contested number here. What fraction of the hardware roofline (memory bandwidth or tensor-core FLOPs, whichever binds) your serving stack actually achieves — kernel efficiency, scheduling gaps, sampling, CPU overhead. 50% reproduces published MLPerf Llama-70B H100 results within ~20%; tuned TensorRT-LLM can exceed it, an untuned stack falls well below. Calibrate it from your own deployment in section 5.
5. Calibrate from Real Deployment
From your own vLLM/SGLang/TensorRT-LLM deployment: sustained output tokens/sec on ONE replica with the topology shown above (same model, precision, GPU and prompt/response lengths), and how many requests were in flight. This back-solves serving efficiency from real data and remembers it in this browser.

Memory Constraint — GPUs per Replica

Throughput Constraint — Replicas Needed

Override

This standalone sketch doesn't push the count into a cost model — the unified planner does that handoff.