Pick a model + a fine-tuning use case → get a recommended GPU count and expected training time, with the full math shown.
Runs entirely client-side. Sizing is a planning heuristic — not a capacity guarantee. Override anything.
1. Model
This model routes each token to a subset of experts (Mixture-of-Experts). "Active experts" computes training FLOPs off only the experts actually touched per token — the standard MoE training-compute convention (matches how DeepSeek-V3/Mixtral report their own training compute). "Full / all experts" instead assumes every expert is touched every token, a conservative upper bound. Memory sizing always accounts for the full expert set either way — all experts' weights/optimizer states must stay resident regardless of this toggle.
Mixed-precision training keeps FP32 master weights and Adam state either way (16 bytes/param of training state); this sets the compute rate and the frozen-weight size under LoRA.
Off = the standard 6 × params × tokens. On adds the attention work (12 × heads × head-dim × attended tokens per layer: scores and value mixing, forward + backward) — material at long sequence lengths; sliding-window layers only count their window.
ZeRO-3/FSDP shards weights, gradients, and optimizer state evenly across every GPU in the job — how virtually all production training actually runs today. Legacy assumes a fixed-size group sized to fit one full copy, then cloned whole for more compute (can overstate GPU count when your compute need lands just past a clone boundary).
2. Training Job
3. GPU
4. Advanced / Contested Assumptions
Applied on top of Korthikanti et al. 2022's activation-memory formula. 1.0x = paper baseline; flash-attention-style kernels typically land under 1.0x, older/naive implementations can exceed it.
The most contested number here. Same role as inference's serving efficiency.
Discount for cross-GPU communication as you add GPUs. Under ZeRO-3/FSDP this covers real added traffic (parameter all-gather on top of the gradient reduce, ~1.5x baseline comm volume per Rajbhandari et al. 2019) — real large runs report 55-60% sustained MFU with good overlap, down to ~7% MFU regression scaling 8→512 GPUs (Zhao et al. 2023).