Ray Dataset Out of Memory

Ray Dataset OOM occurs when the dataset pipeline uses more memory than available in Ray object store or worker memory.

Ray Dataset Out of Memory is a Infrastructure failure seen during ML training runs. Ray Dataset OOM occurs when the dataset pipeline uses more memory than available in Ray object store or worker memory. Common tags: Ray Dataset, Oom, Infrastructure, Object Store.

Object store memory exhausted by dataset blocks. Worker memory exhausted during transformation. Too many blocks cached in object store. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Ray Dataset pipeline fails with OOM
• Workers crash with OOM during data loading
• Training stalls waiting for data

• Ray Data with large datasets
• Ray Data with complex transformations
• Ray Data with many workers

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: Ray actor died with OOM during data loading
• ✓Verified signal present: Object store memory exhausted
• ✓Verified signal present: Worker OOM during dataset transformation
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

• Object store memory exhausted by dataset blocks
• Worker memory exhausted during transformation
• Too many blocks cached in object store