Ray Actor Resource Reservation Causing GPU Starvation from Occupied CPU Slots

Anyscale users experienced GPU starvation when CPU-heavy Ray actors consumed all available CPU slots on GPU nodes, preventing GPU-dependent actors from ever being scheduled. A SpectrogramExtractor with concurrency=8 requested 1 CPU per actor, and Ray distributed 4 of these to the GPU node's 4 CPU slots. The GPU actor was permanently blocked waiting for a CPU slot on the only node with a GPU, stalling the pipeline for nearly an hour.

Ray Actor Resource Reservation Causing GPU Starvation from Occupied CPU Slots is a Infrastructure failure seen during ML training runs. Anyscale users experienced GPU starvation when CPU-heavy Ray actors consumed all available CPU slots on GPU nodes, preventing GPU-dependent actors from ever being scheduled. A SpectrogramExtractor with concurrency=8 requested 1 CPU per actor, and Ray distributed 4 of these to the GPU node's 4 CPU slots. The GPU actor was permanently blocked waiting for a CPU slot on the only node with a GPU, stalling the pipeline for nearly an hour. Common tags: Anyscale, Ray, Actor Scheduling, Cpu Gpu Deadlock.

Ray's default spread scheduling distributes actors across available nodes by CPU count. High-concurrency CPU actors consume all CPU slots on GPU nodes before GPU actors can start. GPU actors require at least 1 CPU + 1 GPU on the same node; with all CPU slots occupied, no GPU actor can ever be scheduled on the GPU node. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• GPU sits idle at 0% utilization for extended periods while CPU actors on the same node occupy slots
• Ray actors with GPU requirements remain in PENDING state indefinitely
• Pipeline makes no progress despite sufficient total cluster resources

• Ray Train / Ray Serve workloads with heterogeneous CPU+GPU actor requirements on shared nodes
• Multi-stage ML pipelines where CPU preprocessing and GPU training run on the same cluster
• Ray clusters without resource isolation between CPU and GPU actors

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: Ray Dashboard shows GPU actor in PENDING state with "resource unavailable" for >10 minutes
• ✓Verified signal present: GPU node shows 100% CPU utilization, 0% GPU utilization
• ✓Verified signal present: Anyscale Data dashboard shows embedding stage with zero output for extended period
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

• Ray's default spread scheduling distributes actors across available nodes by CPU count
• High-concurrency CPU actors consume all CPU slots on GPU nodes before GPU actors can start
• GPU actors require at least 1 CPU + 1 GPU on the same node; with all CPU slots occupied, no GPU actor can ever be scheduled on the GPU node