Docker /dev/shm Exhaustion. NCCL Shared Memory Allocation Failure

Docker containers default to 64MB of /dev/shm, which is far too small for NCCL shared memory inter-process communication. Training crashes with 'posix_fallocate failed: No space left on device'. Denpex detects the /dev/shm exhaustion and recommends the correct Docker configuration.

Docker /dev/shm Exhaustion. NCCL Shared Memory Allocation Failure is a Infrastructure failure seen during ML training runs. Docker containers default to 64MB of /dev/shm, which is far too small for NCCL shared memory inter-process communication. Training crashes with 'posix_fallocate failed: No space left on device'. Denpex detects the /dev/shm exhaustion and recommends the correct Docker configuration. Common tags: Docker, Shm, Shared Memory, Nccl.

Docker sets /dev/shm to 64MB by default, which is too small for NCCL's inter-process shared memory buffers. NCCL allocates shared memory segments for proxy threads, communication channels, and collnet buffers. DataLoader with num_workers > 0 also uses /dev/shm for tensor sharing with the main process. Multiple NCCL communicators (e.g., FSDP sub-communicators) each allocate their own shared memory segments. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• NCCL initialization fails with 'No space left on device' inside Docker containers
• Training works on bare metal but fails inside Docker with shared memory errors
• The container has plenty of disk space but /dev/shm is exhausted

• Docker containers running multi-GPU NCCL training
• Kubernetes pods without /dev/shm size configuration
• Any containerized training with DDP/FSDP and NCCL backend
• Shared memory IPC between DataLoader workers and training processes

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: NCCL WARN posix_fallocate failed: No space left on device
• ✓Verified signal present: ncclShmemOpen failed: cannot allocate shared memory segment
• ✓Verified signal present: df -h /dev/shm shows 64MB (Docker default) and 100% usage
• ✓Verified signal present: Training crashes during NCCL channel or proxy setup
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

NCCL is where a distributed job reports failure, which is not the same as where it failed. The hub lists every common NCCL error next to what it actually indicates, and the environment variables that tell them apart.

• Docker sets /dev/shm to 64MB by default, which is too small for NCCL's inter-process shared memory buffers
• NCCL allocates shared memory segments for proxy threads, communication channels, and collnet buffers
• DataLoader with num_workers > 0 also uses /dev/shm for tensor sharing with the main process
• Multiple NCCL communicators (e.g., FSDP sub-communicators) each allocate their own shared memory segments