CPU Offloading Overhead

CPU offloading trades GPU memory for PCIe bandwidth. Parameters, gradients or optimizer state cross the bus every step, and when the transfer cannot hide behind compute the step becomes bound by the link rather than the GPU.

CPU offloading is a family of memory-reduction techniques that relocate part of a training step's state out of HBM. Three distinct mechanisms are commonly meant by the phrase, and they have different costs, so diagnosing one as though it were another is the usual reason a fix does not work. PyTorch FSDP CPUOffload keeps sharded parameters and gradients in pinned host memory between uses. The optimizer step then runs on the CPU. DeepSpeed ZeRO-Offload moves optimizer state, and optionally parameters under ZeRO-3, to host RAM. The optimizer update is executed by a CPU implementation of Adam. DeepSpeed NVMe offload extends the same idea to a local SSD, adding a filesystem and block-device hop underneath the PCIe transfer. In all three the GPU stalls whenever the tensor it needs is not resident. That stall is the failure being diagnosed. It presents as low GPU utilization with healthy memory headroom, which is the opposite of the pattern engineers expect from a memory-reduction feature.

A training step has a fixed amount of arithmetic and, once offloading is enabled, a fixed amount of data movement. The movement can only be free if it overlaps with compute, and there are structural reasons it often cannot. The first is ordering. Parameters must arrive before the layer that consumes them executes, and gradients must leave before the buffer is reused. Prefetch depth is bounded by how much memory is left to stage into, which is small precisely because memory pressure is why offloading was enabled. The second is the link. Host transfers traverse PCIe. A Gen4 x16 link runs at 16 GT/s per lane with 128b/130b encoding, about 31.5 GB/s per direction in theory and materially less in practice; Gen5 x16 doubles the signalling rate. Compare that with HBM bandwidth measured in terabytes per second and the asymmetry is several orders of magnitude, so a tensor that is cheap to read on device is expensive to fetch across the bus. The third is the optimizer itself. With offloaded optimizer state the Adam update executes on the CPU, so its cost scales with host core count and memory bandwidth rather than with the GPU. A host that is already saturated by data loading has nothing left for it. The fourth is placement. If host memory is allocated on a NUMA node that is not local to the GPU's PCIe root complex, every transfer additionally crosses the inter-socket interconnect. Nothing in the training configuration reveals this; it is a property of process and memory affinity.

• Step time increases sharply after enabling offloading, by far more than the memory saving would suggest is a reasonable trade
• GPU utilization sits low and uneven while GPU memory shows plenty of free headroom, which reads as an underloaded GPU rather than a memory feature working
• Host CPU is pinned near saturation and the data loader begins to fall behind, because the offloaded optimizer step competes with it for the same cores
• Throughput scales worse than linearly when adding ranks per node, since every rank on the node contends for the same PCIe root complex and host memory bandwidth
• NVMe offload shows periodic multi-second stalls that do not correlate with any step boundary in the training loop

• PyTorch FSDP with CPUOffload(offload_params=True), and FSDP2 with CPUOffloadPolicy
• DeepSpeed ZeRO-2 and ZeRO-3 with offload_optimizer.device set to cpu
• DeepSpeed ZeRO-3 with offload_param.device set to cpu or nvme
• Hugging Face Accelerate and Trainer wrapping either of the above through a DeepSpeed or FSDP plugin
• Any multi-GPU node where several ranks share one PCIe root complex or one NUMA domain

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Run the same configuration with offloading disabled for as many steps as memory allows, even a handful, and record median step time for both. If offloading is not the dominant cost the two medians will be close.
• ✓Discard the first ten to twenty steps of every measurement. Compilation, autotuning, allocator growth and cache warming make early steps unrepresentative, and comparing a cold run against a warm one is the most common way this diagnosis is reached incorrectly.
• ✓Profile one warm step and compute the fraction of wall time spent in host-to-device and device-to-host copies. Transfer-bound means that fraction is large, not merely non-zero.
• ✓Check GPU utilization and memory headroom together. Low utilization with substantial free memory is the signature; low utilization with no free memory is a different problem.
• ✓Inspect NUMA and CPU affinity with nvidia-smi topo -m and confirm the process is bound to the node local to its GPU. A mismatch here changes the measurement before any configuration is touched.
• ✓Confirm pinned memory is actually in use. An offload path running on pageable host memory cannot overlap transfers with compute regardless of prefetch settings.
• ✓For NVMe offload, measure the device directly rather than inferring from step time, so a slow disk is not misread as a slow model.

[INFO] [config.py:1006:print] offload_optimizer ............ {'device': 'cpu', 'nvme_path': None, 'buffer_count': 4, 'pin_memory': True, 'pipeline': False, 'ratio': 1.0, 'fast_init': False}
[INFO] [config.py:1006:print] offload_param ................ {'device': 'cpu', 'nvme_path': None, 'buffer_count': 5, 'buffer_size': 100000000, 'max_in_cpu': 1000000000, 'pin_memory': True}
[INFO] [utils.py:781:see_memory_usage] MA 12.44 GB         Max_MA 21.07 GB         CA 24.00 GB         Max_CA 24 GB
[INFO] [stage3.py:128:__init__] Reduce bucket size 500,000,000
Name                       Self CUDA   Self CUDA %   CUDA total   CUDA time avg    # of Calls
Memcpy HtoD (Pageable -> Device)   ---           ---            ---              ---           ---
aten::copy_                        ---           ---            ---              ---           ---
void at::native::vectorized_elementwise_kernel<...>   ---   ---   ---   ---   ---

Use this text as a lookup key in logs and upstream issue trackers. It is not presented as a captured customer log. Confirm the cause from your own preceding events, versions, configuration and the cited references.

Each step here removes one specific reason the transfer cannot be hidden, which is why the order matters more than the individual settings. Measuring warm steps on both configurations turns a subjective slowdown into a ratio, and it is the only step that can tell you offloading is not the cause at all. Correcting NUMA affinity removes a hop that the training configuration cannot express and that no amount of tuning inside the framework will compensate for. Enabling pinned memory is what makes the DMA asynchronous in the first place, so prefetch settings only begin to matter once it is on. Preferring activation checkpointing works because it spends the resource that is not scarce. Recomputation consumes GPU arithmetic, which is idle in exactly the situation being diagnosed, whereas offloading consumes bus bandwidth, which is already the bottleneck. Offloading optimizer state before parameters follows from access frequency. In mixed precision an Adam optimizer keeps two moment estimates plus an fp32 master copy, roughly twelve bytes per parameter against two for a bf16 weight, and that state is read and written once per step. Parameters are needed on every forward and again on every backward, so moving them across the bus costs several times more traffic for a smaller memory saving.

# ---------------------------------------------------------------------------
# 1. Establish the paired baseline. This is the measurement everything else
#    depends on, so run it first and keep both numbers.
#
#    Discard warmup steps. Compilation, autotuning and allocator growth make the
#    first steps unrepresentative, and comparing a cold run to a warm one is the
#    most common way this diagnosis is reached incorrectly.
# ---------------------------------------------------------------------------

import time, statistics, torch

def timed_steps(step_fn, warmup=20, measure=30):
    """Median step time over `measure` warm steps. Returns seconds."""
    for _ in range(warmup):
        step_fn()
    torch.cuda.synchronize()

    samples = []
    for _ in range(measure):
        torch.cuda.reset_peak_memory_stats()
        t0 = time.perf_counter()
        step_fn()
        torch.cuda.synchronize()          # do not time an async launch queue
        samples.append(time.perf_counter() - t0)

    return statistics.median(samples), torch.cuda.max_memory_allocated()

# Run this twice, once with offloading enabled and once without, at identical
# batch size, sequence length and precision. Report both.
#   median_off, peak_off = timed_steps(step)
#   median_on,  peak_on  = timed_steps(step)
# Offloading is worth it only if the memory you bought is worth the time you paid.


# ---------------------------------------------------------------------------
# 2. Attribute the time. Transfer-bound means the copy rows DOMINATE, not merely
#    that they appear. Profile ONE warm step; a trace of a cold step is noise.
# ---------------------------------------------------------------------------

from torch.profiler import profile, ProfilerActivity

for _ in range(20):                        # warm up first
    step()
torch.cuda.synchronize()

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    record_shapes=True,
    profile_memory=True,
) as prof:
    step()
    torch.cuda.synchronize()

# Sort by device time and read the top rows. Memcpy HtoD / DtoH and aten::copy_
# sitting at the top is the confirmation; matmul kernels at the top is not.
print(prof.key_averages().table(sort_by="self_cuda_time_total", row_limit=25))


# ---------------------------------------------------------------------------
# 3. Same question at the system level, including the gaps a Python profiler
#    cannot see. Run for a bounded window, not the whole job.
# ---------------------------------------------------------------------------
#
#   nsys profile \
#     --trace=cuda,nvtx,osrt \
#     --sample=cpu \
#     --duration=60 \
#     --output=offload_trace \
#     python train.py
#
# In the timeline, look for gaps on the CUDA compute row that line up with
# activity on the memory row. That alignment is the stall being diagnosed.


# ---------------------------------------------------------------------------
# 4. Placement. A non-local NUMA node adds an inter-socket hop to every
#    transfer, and nothing in the training config reveals it.
# ---------------------------------------------------------------------------
#
#   nvidia-smi topo -m          # read the CPU Affinity and NUMA Affinity columns
#   numactl --hardware          # confirm the node layout
#   nvidia-smi --query-gpu=index,pcie.link.gen.current,pcie.link.width.current \
#              --format=csv     # confirm the link is running at full width
#
# Then launch each rank bound to the node local to its GPU, for example:
#   numactl --cpunodebind=0 --membind=0 python train.py
#
# Host saturation check, run WHILE training:
#   mpstat -P ALL 1 10          # per-core utilization during the optimizer step
#   numastat -p $(pgrep -f train.py)   # is memory actually on the local node

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

DeepSpeed changes when parameters, gradients and optimizer state are created, partitioned, gathered and offloaded. The hub separates ZeRO, memory, checkpoint and pipeline failures by lifecycle phase.

• Per-step data movement exceeds what the step's compute can hide. The GPU idles waiting for parameters to arrive or gradients to drain, so the step is bound by PCIe rather than by arithmetic.
• Prefetch depth is limited by the free HBM available to stage into. Offloading is usually enabled because that headroom is small, so the mechanism most able to hide the transfer is the one least able to operate.
• With offloaded optimizer state the parameter update runs on the CPU, so its duration is governed by host core count and memory bandwidth and it serializes against data loading on the same cores.
• Host buffers allocated on a non-local NUMA node force every transfer across the inter-socket link in addition to PCIe, which is invisible in the training configuration and is a property of process affinity.
• Unpinned host memory prevents true asynchronous DMA, so transfers are staged through an intermediate copy and cannot overlap with compute even when ordering would allow it.
• NVMe offload adds filesystem and block-layer latency beneath the PCIe transfer, and without a correctly configured asynchronous I/O backend the queue depth is too shallow to reach the device's rated throughput.