How is LLM training compute calculated from model parameters and tokens?

Transformer pretraining compute follows the Chinchilla scaling formula C = 6 * P * D, where P is the number of active non-embedding parameters per token and D is the total number of training tokens. The factor of 6 comes from 2 FLOPs per parameter per token during the forward pass (one multiply and one accumulate across matrix multiplications) plus 4 FLOPs per parameter per token during the backward pass to compute gradients with respect to both activations and weights. For Mixture-of-Experts (MoE) architectures, P represents the active parameters routed per token rather than total stored parameters, while total parameters still govern aggregate VRAM and checkpoint capacity.

What is a realistic MFU for distributed training on H100 clusters?

A realistic sustained Model FLOPs Utilization (MFU) on multi-node NVIDIA H100 SXM5 clusters ranges from 38% to 48% in BF16 when using Megatron-LM or TorchTitan with 3D parallelism (tensor, pipeline, and data parallelism) over 400 Gbps InfiniBand or RoCEv2. Small single-node or 2-node FSDP runs can reach 50% to 54% MFU, while 4,096+ GPU jobs or sparse Mixture-of-Experts runs with heavy all-to-all expert routing frequently operate between 30% and 38% MFU due to pipeline bubbles, collective communication latency, and activation recomputation.

How much does it cost to train a 70B parameter model?

Training a dense 70B parameter model on 15 trillion tokens (the Llama 3 70B over-training regime) requires 6 * 70e9 * 15e12 = 6.3e24 FLOPs (6,300 ExaFLOPs). On an H100 SXM5 cluster running at 40% MFU (roughly 395.6 effective TFLOP/s per GPU), this consumes approximately 4.42 million GPU-hours. At reserved cloud pricing of $2.20 to $2.85 per H100 GPU-hour, raw compute rental costs between $9.7M and $12.6M. A Chinchilla-optimal 70B run on 1.4 trillion tokens requires roughly 413,000 H100 GPU-hours, costing $910K to $1.18M.

Why does Model FLOPs Utilization differ from Hardware FLOPs Utilization (HFU)?

Model FLOPs Utilization (MFU) measures the ratio of the theoretical minimum mathematical operations required for a forward and backward pass (6 * P * D) against the peak theoretical throughput of the hardware. Hardware FLOPs Utilization (HFU) includes extra floating-point operations actually executed on the GPU, most notably selective or full activation checkpointing (recomputing forward activations during the backward pass to save VRAM). Because recomputation adds up to 33% more forward-pass math, a training job can exhibit 52% HFU on hardware counters while delivering only 39% true MFU toward training progress.

How does GPU memory bandwidth and interconnect affect training time?

While large GEMMs (matrix multiplications) are compute-bound on Tensor Cores, normalization layers (RMSNorm), gated activations (SwiGLU), optimizer steps, and attention mechanisms are memory-bandwidth-bound on HBM3/HBM3e. Upgrading from H100 (3.35 TB/s) to H200 (4.8 TB/s) or B200 (8.0 TB/s) reduces kernel launch stalls and allows larger micro-batches without activation offloading. Across nodes, interconnect bandwidth (900 GB/s NVLink intra-node and 400 to 800 Gbps InfiniBand inter-node) dictates how quickly Tensor Parallel all-reduce and FSDP/ZeRO all-gather collectives complete; stepping outside an NVLink domain for Tensor Parallelism can drop MFU by 15 to 25 percentage points.

Free developer tool · no signup · no API key

LLM Training Time, MFU & Cluster Burn Calculator

Model wall-clock pretraining duration, sustained cluster PFLOP/s, mixed-precision memory footprint, and cloud rental burn across NVIDIA H100, H200, B200, A100, and L40S clusters. Built for capacity planners and ML systems engineers sizing dense and Mixture-of-Experts (MoE) training runs.

Reference Model Architecture Presets

Load a frontier pre-training configuration or customize parameters, dataset tokens, and cluster topology.

Chinchilla Scaling (C = 6 * P_active * D)

Llama 3 70B: Dense 70B parameter frontier enterprise model trained on 15 trillion tokens.

1. Model & Dataset Parameters

Architecture Type

Total Parameters (Billions)

Determines VRAM & checkpoint size.

Active Params / Token (B)

Equals total parameters for dense models.

Pre-Training Dataset Size (Trillion Tokens)214.3 tokens/active param

Chinchilla compute-optimal ratio is roughly 20 tokens per parameter; modern over-trained runs (Llama 3, DeepSeek) use 100 to 1,800+ tokens per parameter.

2. GPU Cluster & Efficiency

GPU Accelerator Tier

Cluster Size (Total GPUs)256 nodes (8 GPUs/node)

Model FLOPs Utilization (MFU)38%

20% (High comm overhead)38% to 45% (Typical)60% (Peak)

3D parallelism (tensor, pipeline, data) via Megatron-LM or DeepSpeed ZeRO-3 typically achieves 38% to 48% sustained BF16 MFU after all-reduce, pipeline bubble, and activation recomputation overhead.

Cloud Rate ($/hr per GPU)

Default for NVIDIA H100 Tensor Core: $2.60/GPU-hr

Target Schedule (Days)

Used to size the recommended cluster target.

Cluster Sizing & Training Projections

Sustained throughput: 769.68 PFLOP/s (1,832,570 tokens/sec across 2048 GPUs)

Total Training Time

94.7 days

2,273.7 hours (13.5 weeks)

Total GPU Hours

4,656,484

895 tokens/sec per H100

Total Cloud Rental Cost

$12.11M

At $2.60/hr per GPU (excl. storage/egress)

Total Compute Budget

6,300 ZFLOPs

6,300,000 ExaFLOPs (6 * P_active * D)

14-Day Target Cluster

13,859 GPUs

1733 nodes (8x H100 per node)

Checkpoint Cadence

Every 2h

12/day (~980 GB each)

Distributed Mixed-Precision Memory Breakdown (BF16 + FP32 AdamW)

Total Required: 1.34 TB (1,344 GB) / Aggregate Cluster VRAM: 163.84 TB (163,840 GB)

BF16 Weights (2B/param)140 GB

BF16 Gradients (2B/param)140 GB

FP32 AdamW States (12B)840 GB

Activations & Buffers224 GB

Minimum GPUs to fit sharded model state (at 85% usable VRAM ceiling): 20x NVIDIA H100 Tensor CoreEstimated Datacenter Power: 1,792 kW (4,074.4 MWh run total)

Cluster Engineering & Topology Notes

To finish training within 14 days, scale the cluster to 13859 GPUs (1733 nodes of 8 GPUs).

The Chinchilla compute law: how pretraining FLOPs map to wall-clock time

For autoregressive decoder-only Transformers, total pretraining floating-point operations scale linearly with both active model parameters (P) and dataset tokens (D). Following Hoffmann et al. (Chinchilla) and Kaplan et al., the foundational compute approximation is:

Fundamental Transformer Pretraining Compute Equation

C = 6 * P_active * D (and Wall-Clock Seconds = C / (N_gpus * Peak_FLOPs_per_GPU * MFU))

1. Why the Factor of 6? (2F + 4B)

Every linear projection layer performs a matrix multiply-accumulate operation requiring 2 * P FLOPs per token during the forward pass. During backpropagation, computing gradients with respect to both input activations (dL/dX) and layer weights (dL/dW) requires two matrix multiplications of equivalent size, adding 4 * P FLOPs per token for a total of 6 * P * D.

2. Dense vs. Mixture-of-Experts (MoE)

In sparse MoE models such as DeepSeek-V3 (671B total, 37B active) or Mixtral 8x22B (141B total, 39B active), each token is routed to only top-k experts. Training FLOPs scale with active parameters per token (P_active), whereas aggregate cluster VRAM and checkpoint disk footprint scale with total parameters (P_total).

3. Chinchilla Optimal vs. Over-Training

Chinchilla compute optimality prescribes roughly 20 tokens per parameter to minimize training loss for a fixed compute budget. Modern open weights models intentionally over-train (for example, Llama 3 8B on 15T tokens, or 1,875 tokens/param) to shift compute cost from inference serving into one-time pretraining.

Model FLOPs Utilization (MFU) benchmarks across distributed frameworks

Peak datasheet Tensor Core TFLOP/s (such as 989 TFLOP/s BF16 on H100 SXM5) are never achieved end-to-end in distributed training. Pipeline bubbles, collective communication (all-reduce, all-gather, reduce-scatter, all-to-all), activation recomputation, and memory-bound non-matmul kernels reduce effective throughput. Use the production ranges below to calibrate your MFU slider.

Distributed Framework	Parallelism Strategy	Typical Cluster Scale	Sustained BF16 MFU	Primary Scaling Bottleneck
Megatron-LM / Megatron-Core	3D (TP=8, PP=4 to 16, DP + ZeRO-1) + FlashAttention-3	512 to 16,384 GPUs	41% to 48%	Pipeline parallel schedule bubbles (1F1B / interleaved) and intra-node NVLink TP all-reduce
PyTorch FSDP2 / TorchTitan	2D/3D (TP=8 intra-node, FSDP inter-node) + Async TP	64 to 2,048 GPUs	38% to 45%	All-gather weights before forward/backward and reduce-scatter gradients over InfiniBand
DeepSpeed ZeRO Stage 1 / Stage 2	Data Parallel + Sharded Optimizer States & Gradients	8 to 256 GPUs (Models <= 13B)	46% to 54%	Single-GPU VRAM ceiling; requires full model weights replicated on every rank
DeepSpeed ZeRO Stage 3	Full Parameter, Gradient & Optimizer Sharding across DP	64 to 1,024 GPUs	32% to 40%	High inter-node collective communication volume when hierarchical partitioning (ZeRO++) is disabled
Megatron-MoE / DeepSeek DualPipe	Expert Parallelism (EP) + TP + PP + All-to-All Dispatch	256 to 8,192 GPUs	30% to 38%	Cross-node all-to-all token routing skew, expert load imbalance, and smaller grouped GEMM tiles

Checkpointing and MTBF strategies for 1,000+ GPU clusters

In synchronous distributed training, a single optical transceiver flap, uncorrectable HBM ECC error (Xid 48 / Xid 63), NVLink link fault (Xid 74), or hung NCCL rank halts the entire training job. As cluster size scales past 1,000 GPUs, system Mean Time Between Failures (MTBF) drops from weeks to hours.

64 to 256 GPUs (8 to 32 Nodes)

Expected MTBF

5 to 14 days

Save Cadence

Every 3 to 4 hours

Standard synchronous or threaded host-memory checkpointing to shared Lustre/WEKA storage. Keep 1 warm spare node in the Slurm partition.

512 to 2,048 GPUs (64 to 256 Nodes)

Expected MTBF

18 to 48 hours

Save Cadence

Every 1 to 2 hours

Asynchronous distributed checkpointing (PyTorch DCP or Megatron async_save) staging to pinned CPU RAM first so GPU compute resumes in under 5 seconds.

4,096 to 16,384+ GPUs (512 to 2,048+ Nodes)

Expected MTBF

3 to 12 hours

Save Cadence

Every 30 to 45 minutes

Two-tier in-memory peer replication plus async NVMe burst-buffer flushes, automated Xid/ECC node cordoning, and elastic torchrun/Slurm requeue without manual triage.

Young-Daly optimal checkpoint interval formula

To minimize total wasted GPU-hours from both checkpoint write pauses and post-crash rework, set your checkpoint interval tau using the Young-Daly first-order approximation: tau = sqrt(2 * delta * M), where delta is the time required to commit a checkpoint and M is the cluster MTBF. Adopting asynchronous distributed checkpointing reduces effective delta from minutes to seconds, allowing 30-minute snapshots on 4,096-GPU runs with less than 1.5% throughput overhead.

Frequently asked questions

How is LLM training compute calculated from model parameters and tokens?: Transformer pretraining compute follows the Chinchilla scaling formula C = 6 * P * D, where P is the number of active non-embedding parameters per token and D is the total number of training tokens. The factor of 6 comes from 2 FLOPs per parameter per token during the forward pass (one multiply and one accumulate across matrix multiplications) plus 4 FLOPs per parameter per token during the backward pass to compute gradients with respect to both activations and weights. For Mixture-of-Experts (MoE) architectures, P represents the active parameters routed per token rather than total stored parameters, while total parameters still govern aggregate VRAM and checkpoint capacity.
What is a realistic MFU for distributed training on H100 clusters?: A realistic sustained Model FLOPs Utilization (MFU) on multi-node NVIDIA H100 SXM5 clusters ranges from 38% to 48% in BF16 when using Megatron-LM or TorchTitan with 3D parallelism (tensor, pipeline, and data parallelism) over 400 Gbps InfiniBand or RoCEv2. Small single-node or 2-node FSDP runs can reach 50% to 54% MFU, while 4,096+ GPU jobs or sparse Mixture-of-Experts runs with heavy all-to-all expert routing frequently operate between 30% and 38% MFU due to pipeline bubbles, collective communication latency, and activation recomputation.
How much does it cost to train a 70B parameter model?: Training a dense 70B parameter model on 15 trillion tokens (the Llama 3 70B over-training regime) requires 6 * 70e9 * 15e12 = 6.3e24 FLOPs (6,300 ExaFLOPs). On an H100 SXM5 cluster running at 40% MFU (roughly 395.6 effective TFLOP/s per GPU), this consumes approximately 4.42 million GPU-hours. At reserved cloud pricing of $2.20 to $2.85 per H100 GPU-hour, raw compute rental costs between $9.7M and $12.6M. A Chinchilla-optimal 70B run on 1.4 trillion tokens requires roughly 413,000 H100 GPU-hours, costing $910K to $1.18M.
Why does Model FLOPs Utilization differ from Hardware FLOPs Utilization (HFU)?: Model FLOPs Utilization (MFU) measures the ratio of the theoretical minimum mathematical operations required for a forward and backward pass (6 * P * D) against the peak theoretical throughput of the hardware. Hardware FLOPs Utilization (HFU) includes extra floating-point operations actually executed on the GPU, most notably selective or full activation checkpointing (recomputing forward activations during the backward pass to save VRAM). Because recomputation adds up to 33% more forward-pass math, a training job can exhibit 52% HFU on hardware counters while delivering only 39% true MFU toward training progress.
How does GPU memory bandwidth and interconnect affect training time?: While large GEMMs (matrix multiplications) are compute-bound on Tensor Cores, normalization layers (RMSNorm), gated activations (SwiGLU), optimizer steps, and attention mechanisms are memory-bandwidth-bound on HBM3/HBM3e. Upgrading from H100 (3.35 TB/s) to H200 (4.8 TB/s) or B200 (8.0 TB/s) reduces kernel launch stalls and allows larger micro-batches without activation offloading. Across nodes, interconnect bandwidth (900 GB/s NVLink intra-node and 400 to 800 Gbps InfiniBand inter-node) dictates how quickly Tensor Parallel all-reduce and FSDP/ZeRO all-gather collectives complete; stepping outside an NVLink domain for Tensor Parallelism can drop MFU by 15 to 25 percentage points.

Zero-telemetry cluster diagnostics

Protect your cluster burn rate from silent NCCL hangs and Xid faults

When a 1,024-GPU training job stalls at $2,500 per hour, manual grep across thousands of rank logs wastes critical compute budget. Use the Denpex GPU Log Analyzer in your browser or install our Open VSX / VS Code extension to pinpoint the root-cause rank, hardware Xid code, or C++ ABI mismatch offline in seconds.

Open GPU Log Analyzer Install from Open VSX

Pair your training time estimate with exact VRAM sizing, Slurm job scripts, and log triage utilities.

View all developer tools

Incident Triage