Ray Data Auto-Scaling Failure from Resource Shape Mismatch Between Requested and Available

Ray Data auto-scaling fails when operator resource requirements exceed available cluster resources, throwing ActorUnschedulableError. The specific case documented by Anyscale involved a MapBatches operator with concurrency=2, each task requesting 1 CPU and 1 GPU, totaling 2 CPUs and 2 GPUs, but the cluster did not have enough resources to satisfy the combined GPU+CPU placement constraint.

Ray Data Auto-Scaling Failure from Resource Shape Mismatch Between Requested and Available is a Infrastructure failure seen during ML training runs. Ray Data auto-scaling fails when operator resource requirements exceed available cluster resources, throwing ActorUnschedulableError. The specific case documented by Anyscale involved a MapBatches operator with concurrency=2, each task requesting 1 CPU and 1 GPU, totaling 2 CPUs and 2 GPUs, but the cluster did not have enough resources to satisfy the combined GPU+CPU placement constraint. Common tags: Anyscale, Ray, Autoscaling, Resource Shape.

Ray Data operator resource requirement (GPU:1, CPU:1) does not match any available node shape in the cluster or cloud provider. Each MapBatches actor needs 1 GPU + 1 CPU on the same node, but the smallest GPU node has more CPUs than GPUs, making the per-task CPU reservation too small to fit the GPU+CPU constraint efficiently. The combined resource bundle of 2 CPUs + 2 GPUs exceeds any single node's capacity or the cluster cannot add a node with that exact CPU:GPU ratio. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Ray Data pipeline fails with ActorUnschedulableError: The actor is not schedulable
• Auto-scaler reports insufficient compute resources for requested resource bundles
• Job fails 5 minutes after starting as auto-scaler cannot provision nodes matching resource shape

• Ray Data pipelines with MapBatches operators specifying non-standard resource requirements
• GPU clusters where node shapes have specific CPU:GPU ratios (e.g., 4:8 ratio)
• Auto-scaling Ray clusters where new nodes cannot be provisioned with the exact resource bundle requested

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: Error: `ray.exceptions.ActorUnschedulableError: The actor is not schedulable` with resource shape {GPU: 1, CPU: 1}
• ✓Verified signal present: Cluster auto-scaler logs show: "There are insufficient compute resources to run this workload" with specific resource bundle
• ✓Verified signal present: Ray Dashboard shows pending actors with resource requirements that don't match available node shapes
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

• Ray Data operator resource requirement (GPU:1, CPU:1) does not match any available node shape in the cluster or cloud provider
• Each MapBatches actor needs 1 GPU + 1 CPU on the same node, but the smallest GPU node has more CPUs than GPUs, making the per-task CPU reservation too small to fit the GPU+CPU constraint efficiently
• The combined resource bundle of 2 CPUs + 2 GPUs exceeds any single node's capacity or the cluster cannot add a node with that exact CPU:GPU ratio