Network Storage Volume Causing Process Hangs on H100 GPU Nodes

RunPod discovered an issue affecting pods using volume disk or network storage in their CA-MTL-1 region. When executing commands on storage-backed files, processes would hang indefinitely even though files were created successfully. The issue primarily impacted H100 GPUs and some A40 GPUs, requiring targeted filesystem updates on affected machines.

Network Storage Volume Causing Process Hangs on H100 GPU Nodes is a Infrastructure failure seen during ML training runs. RunPod discovered an issue affecting pods using volume disk or network storage in their CA-MTL-1 region. When executing commands on storage-backed files, processes would hang indefinitely even though files were created successfully. The issue primarily impacted H100 GPUs and some A40 GPUs, requiring targeted filesystem updates on affected machines. Common tags: Runpod, Network Storage, Process Hang, Nfs.

Filesystem driver bug triggered by specific kernel/storage combination on H100 node images. Network storage metadata operation race condition on concurrent file access. Filesystem hang propagates through dataloader pipeline, stalling all GPU workers on the node. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• File operations on network storage hang indefinitely without returning an error
• GPU compute appears healthy (normal utilization, no errors) but training stalls on I/O operations
• Storage writes succeed (file created) but subsequent operations on the file hang

• GPU pods using network-attached storage volumes
• H100 GPU nodes running dataloading-heavy training workloads
• Multi-node training pipelines reading checkpoints or datasets from shared storage

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: Pod logs show process hanging at file I/O operations (open, read, write, stat)
• ✓Verified signal present: strace on the hanging process shows syscall blocked in D state (uninterruptible sleep)
• ✓Verified signal present: GPU utilization drops to 0% during hangs while training loop calls dataloader
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

• Filesystem driver bug triggered by specific kernel/storage combination on H100 node images
• Network storage metadata operation race condition on concurrent file access
• Filesystem hang propagates through dataloader pipeline, stalling all GPU workers on the node