What We Catch
Kubernetes
- Pod eviction failures.
- PVC binding failures.
- CSI driver errors.
- OPA admission webhook denials.
- PriorityClass mismatches.
- TopologySpreadConstraint violations.
- Pod SchedulingGate blocking eviction.
Volcano
podgroups.scheduling.volcano.sh incomplete.queues.scheduling.volcano.sh priority inversion.policies.scheduling.volcano.sh mismatch.
Slurm
- squeue / sdrain failure.
- scontrol update failures.
- Spank plugin errors.
TorchElastic
- etcd v3 / c10d Store errors.
- Min workers / max workers / restart count drift.
Nomad
- Allocation restarts.
- Client disconnection.
Ray / KubeRay
- Actor death rate.
- Object store pressure.
- GCS heartbeat loss.
- Autoscaler events.
Rules
| Rule | Trigger |
|---|
| K8S_POD_QUEUED_TOO_LONG | > 5m pending |
| VOLCANO_PG_INCOMPLETE | pg phase = Incomplete |
| SLURM_DRAIN_FAIL | POST /nodes/{name} returns error |
| TORCHELASTIC_NO_MIN_WORKERS | below min |
| RAY_ACTOR_DEATH_RATE_HIGH | > N / minute |
| RAY_GCS_HEARTBEAT_LOST | gcs down |
| RAY_PLACEMENT_GROUP_RESHAPE | reshape > X GB |
Common Patterns
K8s: pod pending > 5m
- Check
kubectl describe pod for the event. - Cross with TopologySpreadConstraint.
- If Node-removal happened recently, the orchestrator may have just completed draining.
Volcano: podgroup incomplete
- Usually means one rank failed to launch.
- Denpex incident points to the affected rank.
Workflow
denpex incident list --tenant=acme
denpex incident show <id>
denpex incident approve <id> --action=RerouteJob
See Also
docs/operator-guide.md.docs/kubernetes-deployment-guide.md.docs/slurm-deployment-guide.md.