OOMKilled Containers Without Clear Attribution

Kubernetes OOM kills report the process name (e.g., 'python' or 'java') rather than the pod name, making it highly difficult to trace which specific distributed training job caused a cluster-wide memory exhaustion.

OOMKilled Containers Without Clear Attribution is a Infrastructure failure seen during ML training runs. Kubernetes OOM kills report the process name (e.g., 'python' or 'java') rather than the pod name, making it highly difficult to trace which specific distributed training job caused a cluster-wide memory exhaustion. Common tags: Kubernetes, Oom, Memory, Cgroup.

The Linux OOM killer operates at the cgroup level and logs the binary name, lacking K8s context. Memory leaks in Python data loaders (e.g., pinned memory not freed) or un-garbage-collected tensors. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Nodes experience OOM events but the root cause pod is unknown
• Training jobs are killed arbitrarily because a neighboring pod leaked memory
• Standard Prometheus metrics don't link the OOM syslog event to the K8s pod

• Kubernetes (EKS, GKE, AKS) clusters running multi-tenant ML workloads
• Jobs with high data loader memory requirements or memory leaks

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: dmesg shows 'Out of memory: Killed process' with name 'python'
• ✓Verified signal present: kubectl describe pod shows 'OOMKilled' but no specific memory growth pattern
• ✓Verified signal present: System memory usage creeps up over days/weeks without hitting pod limits
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

• The Linux OOM killer operates at the cgroup level and logs the binary name, lacking K8s context
• Memory leaks in Python data loaders (e.g., pinned memory not freed) or un-garbage-collected tensors