DeepSpeed changes when model parameters, gradients and optimizer states are created, partitioned, gathered and offloaded. That is why a failure that looks like an ordinary CUDA OOM can actually come from constructing the model before ZeRO partitioning, while a process killed with no traceback can be host RAM exhaustion caused by CPU offload.
Identify the lifecycle phase first: initialization, forward or backward, parameter gather, optimizer step, or checkpoint save. Then compare the resolved DeepSpeed configuration with the installed DeepSpeed and PyTorch versions. A config copied from another release can validate differently or select a different execution path.
For distributed failures, preserve the first distinct error from every rank. The final launcher exception and NCCL timeout usually describe the surviving ranks after one participant failed. They do not identify which rank or memory tier caused the original event.
CUDA out of memory during DeepSpeed engine initialization
The selected ZeRO stage, model construction path, activation footprint and offload settings still exceed a device memory boundary. Enabling ZeRO does not make every allocation partitioned automatically.
First action: Run python -m deepspeed.env_report, record nvidia-smi memory before initialization, and confirm the model is constructed under deepspeed.zero.Init when using ZeRO stage 3.
The resolved configuration contains an invalid field, value or combination for the installed DeepSpeed release. Framework wrappers may also generate the final config from auto values rather than use the JSON exactly as written.
First action: Run python -m deepspeed.env_report, print the resolved configuration passed to deepspeed.initialize, and compare the first validation field with the configuration reference for that installed version.
AssertionError: Cannot partition a param in flight
ZeRO stage 3 was asked to repartition a parameter while a gather using that parameter was still active. Checkpoint saves at a non-boundary step and unsupported overlapping module use can create this state.
First action: Align the checkpoint interval with gradient_accumulation_steps, then reproduce at a clean optimizer-step boundary and record the installed DeepSpeed version.
A previous forward path left ZeRO parameter fetches active when backward began. Conditional branches, reused modules and dynamic RLHF or search graphs are common triggers because ranks may traverse different module paths.
First action: Set TORCH_DISTRIBUTED_DEBUG=DETAIL, compare the forward module order across ranks, and reproduce with one fixed branch before changing collective timeouts.
Process killed with return code -9 and no traceback during ZeRO offload
The operating system killed a rank after CPU or NVMe offload exhausted host memory or pinned-memory capacity. There is no Python traceback because the process did not get an opportunity to raise one.
First action: Run dmesg -T | grep -i -E "killed process|out of memory" and inspect host memory per rank, then test with pin_memory disabled in both offload sections.
Repeated floating-point overflow forced dynamic loss scaling down to its configured floor and training still could not produce a finite step. This is a numeric stability failure, not a GPU capacity error.
First action: Inspect gradients for the first non-finite value and, on supported hardware, run one controlled step with bf16 enabled and fp16 disabled in the DeepSpeed config.
RuntimeError during ZeRO-3 allgather with parameters of multiple dtypes
The model contains parameters in more than one dtype and the installed DeepSpeed path cannot gather them together correctly. It is a version and model-dtype interaction, not a generic fabric failure.
First action: Print each parameter dtype with python before wrapping the model, then compare a single-dtype control or upgrade to a release that supports ZeRO stage 3 mixed-dtype allgather.
PipelineModule stops progressing when microbatch input shapes vary
Pipeline stages disagreed about the tensor shape expected for a microbatch, so matching sends and receives no longer progressed. Variable sequence shapes expose this when the selected engine path assumes static shapes.
First action: Bucket or pad one reproduction to a fixed shape and set TORCH_DISTRIBUTED_DEBUG=DETAIL. If progress returns, upgrade or keep shape bucketing as the compatibility boundary.
CUDA OOM during AutoModel.from_pretrained under ZeRO-3
The full model was materialized on each rank before ZeRO stage 3 could partition it. The steady-state partition may fit while the temporary unpartitioned construction peak does not.
First action: Construct the model inside deepspeed.zero.Init, or initialize HfDeepSpeedConfig before AutoModel.from_pretrained, then compare nvidia-smi memory during the same load.
Training stops making progress at a DeepSpeed checkpoint save step
The asynchronous checkpoint path stopped draining work, so the training loop waited indefinitely without a final exception. Storage latency, queue pressure and engine defects are separate competing causes.
First action: Capture all rank stacks and storage latency, then repeat one checkpoint with the synchronous engine. A clean synchronous save isolates the decoupled path from storage correctness.
Errors not listed here exist. Rather than guess at their meaning, check Official DeepSpeed ZeRO documentation, which is the authority for the strings above.
Environment variables worth knowing
Most of these are diagnostics rather than fixes. If one makes a failure disappear, it has told you where the fault is, not removed it.
Environment variables and what each one does.
Variable
What it does
python -m deepspeed.env_report
Prints the installed DeepSpeed, PyTorch, CUDA and extension state. Capture it before changing versions so the failing environment remains reproducible.
TORCH_DISTRIBUTED_DEBUG=DETAIL
Adds rank and collective detail for divergent forward paths, hangs and mismatched work. Use it for a bounded reproduction because it increases log volume.
NCCL_DEBUG=INFO
Shows communicator and transport setup when a DeepSpeed failure crosses ranks. The first DeepSpeed or CUDA error before the NCCL cascade remains the priority.
zero_optimization.stage
Controls what ZeRO partitions. Stage 1 partitions optimizer state, stage 2 adds gradients, and stage 3 also partitions parameters.
offload_optimizer.device
Moves optimizer state and work to CPU or NVMe. It trades device memory for host memory, bandwidth and storage pressure rather than eliminating memory use.
stage3_gather_16bit_weights_on_model_save
Controls whether partitioned weights are gathered for a conventional model save. Gathering creates a real memory and communication peak.
Frequently asked questions
Why can DeepSpeed still run out of GPU memory with ZeRO stage 3?
ZeRO partitions model states, but activations, temporary gathers, kernels and a model constructed outside deepspeed.zero.Init can still create an unpartitioned peak. Record memory by lifecycle phase instead of treating the final OOM as a steady-state measurement.
Why did a DeepSpeed rank exit with code -9 and no Python traceback?
Exit code -9 commonly means the operating system sent SIGKILL, often after the host memory cgroup was exhausted by CPU offload or pinned buffers. Check the host kernel and cgroup events before investigating Python, because a killed process cannot emit its own traceback.
Is a DeepSpeed NCCL timeout usually an NCCL bug?
Usually not. One rank often failed earlier during CUDA work, parameter gather, host-memory pressure or checkpoint I/O, while the remaining ranks eventually timed out in a collective. Compare all rank logs and preserve the earliest distinct error rather than the most repeated one.
What should I record with a DeepSpeed error report?
Include the resolved config, deepspeed.env_report output, full launcher command, model construction order, ZeRO stage, offload targets, gradient accumulation, checkpoint step, first error from every rank and host OOM or NVIDIA Xid evidence.
How do I tell whether a DeepSpeed fix is real?
Repeat the same production-shaped input while changing one variable, then require the initiating signature to disappear across every rank and one complete checkpoint or optimizer boundary. A single successful retry after changing versions, batch size and offload together does not identify the cause.
Knowing the error is not knowing the cause
Which rank failed first, whether the checkpoint is safe to resume from, and whether this is your code or the hardware. Paste the log and Denpex answers all three, free and without an account.