Skip to content
Free developer tool · no signup · runs locally in browser

PyTorch DataLoader /dev/shm & Bus Error Sizer for Docker and Kubernetes

Size your container shared memory before training crashes with RuntimeError: DataLoader worker (pid XXX) is killed by signal: Bus error. Model your per-GPU batch size, tensor shape, num_workers, and prefetch_factor to generate ready-to-use Docker, Compose, and Kubernetes emptyDir configurations.

DataLoader Shared Memory Calculator

Calculate required Docker and Kubernetes /dev/shm allocations for PyTorch training workloads. Avoid SIGBUS Bus errors caused by IPC ring buffer exhaustion during multi-worker data loading.

Workload Presets

Configuration

CRITICAL BUS ERROR RISK> 64 MB Default
Recommended --shm-size
8 GB
Safe allocation for container deployment
Total Active IPC Memory
6350.4 MB
Across all 8 GPUs and workers
Single Item Size
< 1 MB
Queue per Worker
73.5 MB
IPC per GPU
588.0 MB

Analysis & Warnings

  • Estimated node /dev/shm usage (6350.4 MB) exceeds the 64 MB default limit in Docker and Kubernetes by 99.2x.
Likely Crash Signature
RuntimeError: DataLoader worker (pid XXX) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory.
# Option 1: Allocate dedicated 8 GB shared memory volume (Recommended)
docker run --gpus all --shm-size=8g your-training-image:latest

# Option 2: Share host IPC namespace directly
docker run --gpus all --ipc=host your-training-image:latest

Why PyTorch multiprocessing uses /dev/shm IPC ring buffers for tensors

When you set num_workers > 0 in torch.utils.data.DataLoader, PyTorch spawns independent Python worker processes to decode files, apply augmentations, and collate samples into contiguous batch tensors. Passing multi-megabyte tensors back to the main training loop over standard OS pipes or sockets would serialize and copy every byte twice, turning CPU IPC into a throughput bottleneck.

1. Zero-Copy Shared Storage Allocation

Instead of pickling raw tensor bytes, worker processes allocate collated batch tensors directly inside POSIX shared memory backed by /dev/shm (via torch_shm_manager and shm_open). Only a lightweight file descriptor handle is sent over the multiprocessing queue to the main process.

2. In-Flight Prefetch Ring Buffer

Each worker stays ahead of the GPU forward pass by pre-populating prefetch_factor batches (defaulting to 2 per worker). On an 8-GPU DDP node with 8 workers per rank, up to 8 × 8 × 2 = 128 full batches reside concurrently inside /dev/shm plus batches currently being consumed by pin_memory threads.

3. Why Exhaustion Triggers SIGBUS

On Linux, ftruncate and mmap succeed lazily when creating the virtual mapping. The actual page allocation occurs when the worker writes batch data into the mapped pointer. If /dev/shm has hit its 64 MB tmpfs ceiling, the kernel cannot fault in a backing page and delivers SIGBUS (Bus error) directly to the worker PID.

Active Node Shared Memory Formula

Active IPC Footprint = GPUs_per_Node × Batch_Size_per_GPU × Sample_Bytes × num_workers × (prefetch_factor + 1) × Safety_Overhead(1.5x)

Default container limits vs real ML workload requirements

Container runtimes inherited their 64 MB /dev/shm default from stateless microservice workloads. Compare that 64 MB ceiling against typical single-GPU and multi-GPU training pipelines below.

Workload ProfileDataLoader ConfigurationActive IPC Queue64 MB Default BehaviorRecommended /dev/shm
Default Docker / Kubernetes ContainerUnconfigured runtime tmpfs mount at /dev/shm64 MB hard ceilingBaseline limitOverride for any multi-worker GPU job
ResNet-50 / ViT-B/16 ImageNet (Single GPU)batch=128, shape=[3, 224, 224], FP32, workers=8, prefetch=2~1.25 GBImmediate SIGBUS crash4 GB
8x H100 DDP Vision-Language Pretraining8 GPUs, batch=64/GPU, shape=[3, 448, 448], BF16, workers=8~10.0 GB across nodeImmediate SIGBUS crash24 GB to 32 GB
3D Medical Segmentation (MONAI / CT Volumes)4 GPUs, batch=2/GPU, shape=[1, 256, 256, 256], FP32, workers=4~12.8 GB across nodeImmediate SIGBUS crash32 GB
Video Diffusion / Spatio-Temporal Clips8 GPUs, batch=4/GPU, shape=[16, 3, 512, 512], FP16, workers=6~18.1 GB across nodeImmediate SIGBUS crash48 GB to 64 GB
Tokenized Text LLM Fine-Tuning (Token IDs Only)8 GPUs, batch=8/GPU, shape=[8192], INT64, workers=4, prefetch=2~50 MB across nodeBorderline (64 MB limit)2 GB to 4 GB

Step-by-step commands for Docker, Podman, and Kubernetes Helm charts

Apply the recommended shared memory allocation from the calculator above using the exact runtime flags for your container orchestrator. Inside any running container, run df -h /dev/shm before launching torchrun to verify that the mount took effect.

1. Docker & Podman CLI

Pass --shm-size or share the host IPC namespace for single-node runs

# Option A: Dedicated isolated tmpfs quota (recommended for multi-tenant hosts)
docker run --gpus all --shm-size=16g --ulimit memlock=-1 -it nvcr.io/nvidia/pytorch:24.12-py3

# Option B: Podman rootless or rootful execution with explicit shm allocation
podman run --device nvidia.com/gpu=all --shm-size=16g -it nvcr.io/nvidia/pytorch:24.12-py3

Always pair --shm-size with --ulimit memlock=-1 when running multi-GPU NCCL workloads so pinned host memory and shared IPC pages are not throttled by default PAM limits.

2. Docker Compose (compose.yaml)

Declare shm_size directly on the training service definition

services:
  pytorch-trainer:
    image: nvcr.io/nvidia/pytorch:24.12-py3
    shm_size: "16gb"
    ipc: private
    ulimits:
      memlock: -1
      stack: 67108864
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Docker Compose v2.x and v3.5+ support top-level shm_size strings. Avoid setting ipc: host on shared staging servers where leftover POSIX segments could leak across builds.

3. Kubernetes Pod & Helm Chart Values

Mount a memory-backed emptyDir volume at /dev/shm with a matching RAM limit

# values.yaml (Helm chart convention for PyTorchJob / RayCluster / Deployment)
shmVolume:
  enabled: true
  sizeLimit: 16Gi

# Resulting Pod template snippet:
# volumes:
#   - name: dshm
#     emptyDir:
#       medium: Memory
#       sizeLimit: 16Gi
# volumeMounts:
#   - name: dshm
#     mountPath: /dev/shm

Kubernetes counts pages written to a medium: Memory emptyDir toward the container's resources.limits.memory budget. Add your sizeLimit (for example 16Gi) on top of your normal host RAM request to prevent kubelet OOMKilled evictions.

Frequently asked questions

Why does PyTorch DataLoader fail with Bus error (pid XXX is killed by signal: Bus error)?
When num_workers is greater than 0, PyTorch DataLoader worker processes collate batches into shared memory backed by /dev/shm so the main training process can read tensors without copying bytes over pipes. If the total prefetched tensor queue exceeds the container's /dev/shm tmpfs quota, Linux fails to allocate backing physical pages when the memory-mapped region is accessed and raises SIGBUS (Bus error) instead of a standard Python MemoryError.
What is the default /dev/shm size in Docker containers?
Docker, containerd, and Podman default to mounting a 64 MB tmpfs filesystem at /dev/shm inside every container unless overridden. While 64 MB is sufficient for lightweight web services, a single 8-GPU PyTorch DistributedDataParallel node with 8 workers per GPU and prefetch_factor=2 routinely stages several gigabytes of collated tensors in shared memory.
How do I increase shared memory in Docker and Docker Compose?
In the Docker or Podman CLI, pass the --shm-size flag (for example, docker run --gpus all --shm-size=16g) or pass --ipc=host to share the host node's /dev/shm directly. In Docker Compose (v2 and v3.5+), set shm_size: '16gb' directly under the service definition.
How do I mount shared memory for PyTorch in Kubernetes?
Kubernetes does not expose a native shmSize field in the Pod spec. To expand /dev/shm in Kubernetes, define a volume using emptyDir with medium: Memory and an explicit sizeLimit (such as 16Gi), then mount that volume at mountPath: /dev/shm in your training container. Note that memory-backed emptyDir consumption counts against your Pod memory limit.
When should I use pin_memory=True and persistent_workers=True in DataLoader?
Enable pin_memory=True whenever transferring tensors from CPU RAM to NVIDIA GPUs so the main process copies data into page-locked (pinned) host memory for asynchronous DMA transfers over PCIe (`non_blocking=True`). Enable persistent_workers=True when training across multiple epochs to keep worker subprocesses alive and avoid tearing down and re-allocating IPC ring buffers at every epoch boundary.
Crash Triage & Verification

Still seeing worker crashes, NCCL timeouts, or OOM kills?

Drop your raw PyTorch, Slurm, or Kubernetes pod log into the Denpex GPU Log Analyzer. It separates /dev/shm SIGBUS failures from cgroup SIGKILL (exit code 137) host RAM exhaustion and pinpoints the first failing rank without uploading proprietary code.

Open GPU Log Analyzer

Related GPU infrastructure tools

Continue diagnosing and sizing your training or inference stack with these zero-signup utilities.

View all developer tools
Triage & Root Cause

GPU & Distributed Training Log Analyzer

Paste raw DataLoader SIGBUS tracebacks, NCCL watchdog timeouts, CUDA OOM logs, or Xid hardware errors for instant offline root-cause diagnosis.

Open tool
Capacity Planning

LLM VRAM & KV Cache Calculator

Estimate model weights, optimizer states, activation memory, and KV cache footprint across FP16, BF16, FP8, and INT4 before launching your job.

Open tool
Environment Setup

CUDA, PyTorch & FlashAttention Resolver

Match your NVIDIA kernel driver, container CUDA runtime, PyTorch wheel index, and FlashAttention C++11 ABI tags with deterministic install commands.

Open tool