Distributed GPU workloads on Ray (such as KubeRay clusters, OpenRLHF, and verl post-training pipelines) separate logical actor placement from physical CUDA memory allocation. When a task stalls or fails, the outer Ray exception often points to scheduler timeouts rather than the underlying GPU or host-RAM boundary.
Start every triage by running ray status to inspect pending placement groups versus physical GPU slots, checking host dmesg for Raylet host-RAM OOM kills, and matching each actor PID to actual VRAM consumption on the node.
The literal string is what you paste into a search bar, so it is the heading. “Class” is a triage category. It is not proof of ownership in your workload.
ray.exceptions.ClusterUnavailableError: Raylet process crashed (OOM killed)
Linux killed the Raylet node daemon or worker object store because plasma /dev/shm and Python host allocations exceeded container or node memory limits, disconnecting GPUs mid-run.
First action: Check dmesg -T | rg -i "killed process|oom-killer" and verify ray status object store memory vs container cgroup limits before raising worker counts.
PlacementGroupCreationError: placement group creation timed out
Ray cannot schedule an entire strict-pack or pack placement bundle because fractional GPU requests, colocated rollout actors, or unavailable nodes leave partial slots blocked.
First action: Run ray status to inspect pending demands vs available GPUs and verify actor num_gpus plus bundle alignment in your OpenRLHF or verl config.
Ray actor task pending indefinitely due to occupied CPU or GPU slots
Parent actor tasks hold GPU reservations while spawning nested tasks waiting on CPU slots, or two colocated actor groups deadlock over remaining fractional GPU reservations.
First action: Inspect ray status and check RAY_memory_monitor_refresh_ms=0 vs explicit --num-gpus and --num-cpus reservations across parent and child actors.
ray.exceptions.RuntimeEnvSetupError: Runtime environment setup timed out
Worker nodes stalled downloading pip wheels, cloning git dependencies, or hitting registry timeouts when building isolated virtualenvs for GPU tasks.
First action: Pre-bake CUDA and framework wheels into the worker container image and check Raylet runtime_env agent logs before raising RAY_RUNTIME_ENV_TEMPORARY_DIR limits.
ray.exceptions.OutOfMemoryError: Ray Dataset or object store exhausted
Plasma object store spilling or Ray Data batch preprocessing exceeded node host memory limits before batches ever reached GPU training actors.
First action: Run ray memory --stats-only and configure smaller batch concurrency or object store spilling paths rather than cutting GPU batch size first.
Why does Ray PlacementGroupCreationError hang even when GPUs appear free in nvidia-smi?
nvidia-smi reports physical GPU devices, whereas Ray schedules against logical resource counters registered when each Raylet joins the cluster. Stale actors, fractional GPU allocations, or strict-pack single-node bundles that exceed one host GPU count keep placement groups pending indefinitely.
Why does Raylet die with ClusterUnavailableError instead of printing a CUDA error?
When Plasma shared memory (/dev/shm) or Python host RAM crosses the pod cgroup ceiling, the Linux OOM killer terminates the Raylet process with SIGKILL. Inspecting host dmesg and container memory limits separates host memory kills from GPU VRAM exhaustion.
How do I prevent colocated vLLM and trainer Ray actors from starving each other in GRPO?
Colocated RL post-training (GRPO, DAPO, PPO) shares HBM across trainer weights, optimizer states, reference logits, and vLLM KV cache. Calculate combined budgets in our GRPO RL VRAM Calculator and reserve fractional num_gpus cleanly.
Should I set RAY_memory_monitor_refresh_ms=0 to stop worker OOM kills?
Disabling the Ray worker memory monitor prevents Ray itself from preempting workers near threshold, but it does not reduce host RAM usage and shifts termination directly to the kernel cgroup OOM killer. Fix object store limits or batch concurrency first.
Knowing the error is not knowing the cause
Which rank failed first, whether the checkpoint is safe to resume from, and whether this is your code or the hardware. Paste the log and Denpex answers all three, free and without an account.