vLLM tensor parallelism fails in a container because shared memory is limited to 64 MB

A container gets 64 MB of shared memory by default. PyTorch and NCCL use that space to move data between the worker processes of a tensor-parallel deployment, so a model that runs on one GPU fails to initialise across several, with an error naming shared memory rather than the model.

A container configuration failure in which the default 64 MB shared memory mount is too small for the inter-process transfers that tensor-parallel vLLM and NCCL perform, preventing the engine from initialising across multiple GPUs.

The shared memory default is a property of the container runtime, chosen without reference to collective communication. vLLM's workers exchange tensors through that mount and NCCL creates segments in it during setup, so a limit that is generous for a web service is far too small here. Because a single-GPU deployment never touches the segment, the constraint stays hidden until the day someone scales across devices.

• The same image and model work with one GPU and fail with two or more
• The failure occurs during initialisation, before any request is served
• The error names NCCL or a shared memory segment, which does not obviously implicate the container runtime
• The deployment works locally and fails on the orchestrator, or the reverse

• Docker containers started without --shm-size or --ipc=host, which receive the 64 MB default
• Kubernetes pods, where the default shared memory size is frequently too small for these containers
• Docker Swarm, which does not support the host IPC option at all
• Any tensor-parallel or pipeline-parallel vLLM deployment, since single-GPU serving does not exercise this path

Apply the following checklist to a small reproduction: each box below is a positive signal that you are looking at this exact failure rather than a sibling in the same taxonomy.

• ✓Run df -h /dev/shm inside the container. A total of 64 MB confirms the default mount is in place and is the limit being hit.
• ✓Start the identical image with tensor parallel size 1. Success on one GPU and failure on more localises the problem to inter-worker communication rather than to the model or weights.
• ✓Re-run with NCCL_DEBUG=INFO and read the first warning. A message naming a shared memory segment points here; one naming peer access points at the transport instead.

NCCL WARN Error while creating shared memory segment /dev/shm/nccl-vzIpS6 (size 9637888)
torch.distributed.DistBackendError: NCCL error: unhandled system error (run with NCCL_DEBUG=INFO for details)
WARNING: /dev/shm has only 67108864 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm.

Timestamps and exact values vary across runs, but the pattern. An info-level start, an early WARN, an ERROR carrying the symptom. Is the actual fingerprint you should alert on. The Denpex platform flags this combination automatically.

Enlarging the mount, or sharing the host's, gives the segment room to be created, so communicator setup completes and the workers can exchange tensors. It is a configuration change to the container rather than to vLLM, which is why no vLLM flag resolves it and why the same image behaves differently across environments.

// Typical pattern:
import torch.optim as optim
optimizer = optim.AdamW(model.parameters(), lr=3e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_training_steps)
for step in range(num_training_steps):
    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()
    optimizer.step()
    scheduler.step()

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.