Xet Storage Migration Workers Filling Disk with Orphaned Temporary Shard Files

During HuggingFace's migration from Git LFS to Xet storage backend, migration worker pods filled their ephemeral disks with orphaned temporary shard files. The shard files were first written to /tmp and then moved to the Xet cache, but /tmp and the cache were on different mount points. The move failed silently, orphan shard files were never cleaned up, and the disk eventually filled, triggering No space left on device errors across the migration fleet.

Xet Storage Migration Workers Filling Disk with Orphaned Temporary Shard Files is a Infrastructure failure seen during ML training runs. During HuggingFace's migration from Git LFS to Xet storage backend, migration worker pods filled their ephemeral disks with orphaned temporary shard files. The shard files were first written to /tmp and then moved to the Xet cache, but /tmp and the cache were on different mount points. The move failed silently, orphan shard files were never cleaned up, and the disk eventually filled, triggering No space left on device errors across the migration fleet. Common tags: Huggingface, Xet, Storage Migration, Disk Pressure.

Shard files were created in /tmp then atomically moved to the cache directory; /tmp and cache sat on different mount points. Cross-filesystem `rename()` silently fails (EXDEV error), falling back to copy+delete which was not implemented in the worker. Orphaned shard files accumulated in /tmp until disk exhaustion; no cleanup mechanism existed for failed temp files. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Migration workers report "No space left on device" errors
• Disk usage on worker pods grows monotonically and never decreases
• Migration throughput drops to zero as workers cannot write new shard files

• Background data migration workers writing to diverse mount points
• Kubernetes pods with separate ephemeral and persistent volume mounts
• Storage backends with atomic move operations spanning filesystem boundaries

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: Pod disk usage at 100% with `df -h` showing /tmp filesystem full
• ✓Verified signal present: Large temporary files remaining in /tmp directory that were intended to be moved to cache mount
• ✓Verified signal present: Profiling shows pods are network and EBS I/O bottlenecked despite apparent CPU capacity
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

• Shard files were created in /tmp then atomically moved to the cache directory; /tmp and cache sat on different mount points
• Cross-filesystem `rename()` silently fails (EXDEV error), falling back to copy+delete which was not implemented in the worker
• Orphaned shard files accumulated in /tmp until disk exhaustion; no cleanup mechanism existed for failed temp files