Pick a model + a use case → get a recommended GPU type & count, with the full math shown. Feeds the TCO calculator's cluster size — not yet wired up.
Runs entirely client-side. Sizing is a planning heuristic based on published model architectures — not a capacity guarantee. Override anything.
1. Model
This model routes each token to a few experts (Mixture-of-Experts). "Routed experts" models what a decode step really reads: the shared layers plus every expert at least one token in the batch was routed to — near the active-param count at batch 1, approaching the full expert set at large batches. "All experts every step" assumes every expert's weights are read and computed every token (an upper bound). Weight memory always holds the full expert set either way.
FP4 formats run on FP4 tensor cores on Blackwell and MI355X. On Hopper/MI300 they still save memory but compute at BF16 rate (weight-only quantization).
Serving stacks (vLLM, SGLang, TensorRT-LLM) quantize the KV cache separately from the weights — it is never stored at 4-bit here.
2. Use Case (Interactive Inference)
Visible answer tokens only — see "Reasoning tokens" below for thinking-mode models.
For thinking/reasoning models (e.g. DeepSeek-R1, o1-style) that generate a hidden chain-of-thought before the visible answer. These tokens are generated autoregressively exactly like any other output token — same compute, bandwidth, and KV cache cost — so leaving this at 0 for a reasoning workload will undersize the cluster. Typical reasoning traces run 1,000–8,000+ tokens depending on task difficulty. Leave at 0 for non-reasoning models.
Time to deliver a complete response (prompt processing + all output tokens). It sets the per-user speed every stream must get: (output + reasoning tokens) ÷ (response time − time-to-first-token). Bigger batches raise total throughput but slow each stream, so this caps how many sessions one replica can serve.
3. GPU
4. Advanced / Contested Assumptions
Fraction of nameplate VRAM actually usable after CUDA context, framework runtime, etc.
Discount for NVLink all-reduce traffic when one replica spans several GPUs.
Only used when a model is too big for one NVLink domain and is split into pipeline stages across nodes (bubbles + inter-node hops).
The most contested number here. What fraction of the hardware roofline (memory bandwidth or tensor-core FLOPs, whichever binds) your serving stack actually achieves — kernel efficiency, scheduling gaps, sampling, CPU overhead. 50% reproduces published MLPerf Llama-70B H100 results within ~20%; tuned TensorRT-LLM can exceed it, an untuned stack falls well below. Calibrate it from your own deployment in section 5.
5. Calibrate from Real Deployment
From your own vLLM/SGLang/TensorRT-LLM deployment: sustained output tokens/sec on ONE replica with the topology shown above (same model, precision, GPU and prompt/response lengths), and how many requests were in flight. This back-solves serving efficiency from real data and remembers it in this browser.
Memory Constraint — GPUs per Replica
Throughput Constraint — Replicas Needed
Override
This standalone sketch doesn't push the count into a cost model — the unified planner does that handoff.