Cluster Shared Storage Slow

Cluster shared storage (NFS, Lustre, GPFS) can be a major bottleneck for distributed training data loading.

Cluster Shared Storage Slow is a Infrastructure failure seen during ML training runs. Cluster shared storage (NFS, Lustre, GPFS) can be a major bottleneck for distributed training data loading. Common tags: Storage, Nfs, Lustre, Gpfs.

Shared storage bandwidth saturated. Too many nodes reading same files. NFS single server bottleneck. Lustre stripe settings not optimized. File count too high for metadata server. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Data loading is the bottleneck
• GPUs are starving for data
• Training is IO-bound

• Distributed training with shared data
• Multi-node training reading same files
• Pretraining on ImageNet/LLM datasets

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: iostat shows high await on storage
• ✓Verified signal present: nfsstat shows slow operations
• ✓Verified signal present: Lustre stripe count not optimal
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

• Shared storage bandwidth saturated
• Too many nodes reading same files
• NFS single server bottleneck
• Lustre stripe settings not optimized
• File count too high for metadata server