Mid-Day Temperature Fluctuations Causing 1-2% Training Throughput Variation

During Llama 3 training on 16,384 H100 GPUs, Meta observed that mid-day temperature fluctuations of 5-10 degrees Celsius caused 1-2% throughput variation across the training cluster. The effect was attributed to GPU dynamic voltage and frequency scaling responding to ambient temperature changes, with the entire cluster's power consumption fluctuating by tens of megawatts throughout the diurnal cycle.

Symptom

nvidia-smi reports GPU clock rate oscillating between base and boost clock on a diurnal schedule

Root cause

GPU boost clock behavior depends on silicon temperature: higher temperature reduces voltage/frequency headroom. Data center cooling capacity lags behind diurnal temperature changes, allowing ambient to rise 5-10C during afternoon. Simultaneous power consumption changes of tens of thousands of GPUs stress the data center power grid.

Recommended fix

nvidia-smi --lock-gpu-clocks=1410,1410

How Denpex helps

Denpex investigates Mid-Day Temperature Fluctuations Causing 1-2% Training Throughput Variation using the evidence you provide or your connected workload collects. Earlier rank, host or application evidence is needed to distinguish an initiating failure from a downstream report.

Mid-Day Temperature Fluctuations Causing 1-2% Training Throughput Variation is a Reliability failure seen during ML training runs. During Llama 3 training on 16,384 H100 GPUs, Meta observed that mid-day temperature fluctuations of 5-10 degrees Celsius caused 1-2% throughput variation across the training cluster. The effect was attributed to GPU dynamic voltage and frequency scaling responding to ambient temperature changes, with the entire cluster's power consumption fluctuating by tens of megawatts throughout the diurnal cycle. Common tags: Meta, Llama3, Temperature, Dvfs.

GPU boost clock behavior depends on silicon temperature: higher temperature reduces voltage/frequency headroom. Data center cooling capacity lags behind diurnal temperature changes, allowing ambient to rise 5-10C during afternoon. Simultaneous power consumption changes of tens of thousands of GPUs stress the data center power grid. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Training throughput varies systematically by 1-2% on a daily cycle correlating with outdoor temperature
• GPU clock rates drop measurably during peak ambient temperature hours
• Cluster-wide power consumption oscillates by tens of megawatts over the diurnal cycle

• Large GPU clusters (>1000 GPUs) in air-cooled data centers
• H100/H200 clusters operating near thermal design power limits
• Training runs spanning multiple days where throughput consistency affects convergence

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: Tokens-per-second metric shows daily sinusoidal pattern with peak at night
• ✓Verified signal present: nvidia-smi reports GPU clock rate oscillating between base and boost clock on a diurnal schedule
• ✓Verified signal present: Data center PUE increases during peak temperature hours reducing total available power budget
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

• GPU boost clock behavior depends on silicon temperature: higher temperature reduces voltage/frequency headroom
• Data center cooling capacity lags behind diurnal temperature changes, allowing ambient to rise 5-10C during afternoon
• Simultaneous power consumption changes of tens of thousands of GPUs stress the data center power grid

nvidia-smi reports GPU clock rate oscillating between base and boost clock on a diurnal schedule

Use this text as a lookup key in logs and upstream issue trackers. It is not presented as a captured customer log. Confirm the cause from your own preceding events, versions, configuration and the cited references.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.