FSDP fails when composed with CPU parameter offloading

Sharding and CPU offloading both move parameters, and each assumes it is the component deciding where a parameter lives. Enabling them together produces a failure during setup or the first step, in a stack that works correctly with either one alone.

A composition failure in which parameter sharding and CPU offloading each claim authority over where a parameter resides, producing a setup-time or first-step error in a configuration where either feature alone succeeds.

Both features exist to reduce resident parameter memory and both work by controlling placement. Sharding splits a parameter and records where each piece lives; offloading moves it to host memory and records that instead. Neither is written to defer to the other, so the composition has no agreed owner for a tensor, and the first thing that resolves placement finds a contradiction.

• Both features are documented and each works by itself
• The failure appears at setup or on the first step rather than under memory pressure
• Offloading was enabled precisely because memory was already tight, so simply removing it is not free
• The error rarely names offloading, so the interaction is not obvious from the message

• Megatron-style FSDP implementations combined with CPU offloading
• Stacks where offloading is configured by a launcher and sharding by the training framework, so neither knows about the other
• Configurations carried over from a single-device memory-saving setup into a sharded one
• Any setup where a parameter's owner is ambiguous between two memory managers

Apply the following checklist to a small reproduction: each box below is a positive signal that you are looking at this exact failure rather than a sibling in the same taxonomy.

• ✓Disable CPU offloading and re-run with sharding unchanged. Success isolates the interaction rather than either feature.
• ✓Disable sharding and re-run with offloading unchanged. Success on both single-feature runs confirms the composition is the problem.
• ✓Check whether the failure occurs at setup or on the first step rather than under sustained memory pressure, which distinguishes a placement disagreement from genuine exhaustion.

Megatron FSDP does not work with CPU_OFFLOADING
RuntimeError raised during setup with both parameter sharding and CPU offloading enabled

Timestamps and exact values vary across runs, but the pattern. An info-level start, an early WARN, an ERROR carrying the symptom. Is the actual fingerprint you should alert on. The Denpex platform flags this combination automatically.

Removing one claimant leaves a single authority and the contradiction disappears. Recovering the memory from activations works because activation memory is managed by a different mechanism entirely, so it does not compete for ownership of parameters. Widening sharding reduces the same footprint offloading was targeting, without a second manager.

// Typical pattern:
import torch.optim as optim
optimizer = optim.AdamW(model.parameters(), lr=3e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_training_steps)
for step in range(num_training_steps):
    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()
    optimizer.step()
    scheduler.step()

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.