Skip to content

PyTorch GPU and distributed errors

PyTorch GPU jobs fail most often at tensor device boundaries, DataLoader shared-memory limits, DistributedDataParallel (DDP) / FSDP collective synchronization, or checkpoint state restoration.

Keep the first worker traceback, tensor devices, and DataLoader or launch settings. A downstream DDP or NCCL timeout frequently hides an earlier exception on a single worker rank.

Working in VS Code or Remote-SSH? Diagnose your log locally with the free VS Code extension. No account or upload required for local diagnosis.

Every common pytorch gpu and distributed error

The literal string is what you paste into a search bar, so it is the heading. “Class” is a triage category. It is not proof of ownership in your workload.

Tensor device mismatch

Application bug
RuntimeError: Expected all tensors to be on the same device

An operator received operands on CPU and CUDA, or across different CUDA device indices, in a multi-GPU or model-parallel step.

First action: Run with CUDA_LAUNCH_BLOCKING=1 and print tensor devices via torch.cuda.current_device() right before the failing operation to verify every tensor and registered buffer moved to the same rank.

Full entry: root cause, fix and prevention
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor

An index tensor or embedding lookup sits on a different GPU or CPU than the indexed tensor.

First action: Check index and mask devices with python -c or use /tools/indices-device-mismatch to move index tensors with register_buffer() across DataLoader and DDP ranks.

Full entry: root cause, fix and prevention
RuntimeError: DataLoader worker (pid N) is killed by signal: Bus error

Linux /dev/shm is too small for inter-process shared-memory tensors between DataLoader workers and the training process.

First action: Check df -h /dev/shm inside the container and raise --shm-size or Kubernetes emptyDir medium: Memory capacity before lowering num_workers.

Full entry: root cause, fix and prevention
RuntimeError: Address already in use (torchrun c10d rendezvous)

Another process owns the rendezvous address and port, or two independent single-node launchers collided on the default master port.

First action: Inspect active listeners with ss -ltnp and set distinct --rdzv-endpoint values or localhost:0 for independent single-node jobs.

Full entry: root cause, fix and prevention
Error(s) in loading state_dict / Optimizer parameter group mismatch

Saved model keys or optimizer parameter groups no longer match the constructed architecture or trainable parameter ordering.

First action: Set TORCH_DISTRIBUTED_DEBUG=DETAIL and run a quick python inspection comparing saved parameter group lengths and rank topology before forcing load flags.

Full entry: root cause, fix and prevention

PyTorch CUDA out of memory

Ambiguous, needs evidence
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate

The caching allocator could not satisfy a tensor request due to total capacity exhaustion or reserved-but-fragmented memory.

First action: Print torch.cuda.memory_summary() and test PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to separate memory fragmentation from true capacity growth in /tools/cuda-oom-triage.

Full entry: root cause, fix and prevention

Errors not listed here exist. Rather than guess at their meaning, check PyTorch Distributed Troubleshooting Guide, which is the authority for the strings above.

Frequently asked questions

Why does a PyTorch tensor device mismatch still appear after calling model.to(device)?
Standalone tensors created in forward, cached masks, DataLoader collate outputs, or unregistered Python lists on nn.Module do not move automatically with model.to(device). Register persistent tensors with register_buffer() and move batch inputs to the rank device explicitly.
Why do DataLoader workers crash with Bus error only inside Docker or Kubernetes?
Default Docker containers allocate only 64 MB to /dev/shm, and Kubernetes pods limit /dev/shm unless you mount an emptyDir backed by Memory. PyTorch worker IPC fills that limit quickly when passing image or token batches.
Does a DDP hang always mean NCCL failed?
No. If one rank raised a Python exception, took a conditional branch missing an all-reduce, or hit CUDA OOM, healthy ranks wait inside DDP backward hooks until the watchdog aborts with an NCCL timeout.
How do I tell PyTorch memory fragmentation apart from actual capacity OOM?
Compare reserved versus allocated bytes inside the CUDA OutOfMemoryError line or torch.cuda.memory_summary(). If gigabytes remain reserved but unallocated, fragmentation is likely and expandable_segments:True can recover contiguous blocks without reducing batch size.

Knowing the error is not knowing the cause

Which rank failed first, whether the checkpoint is safe to resume from, and whether this is your code or the hardware. Paste the log and Denpex answers all three, free and without an account.

Diagnose your logs free