Cgroup Memory Limit

Cgroup memory limits cause OOM kills when training exceeds the container or pod's memory limit.

Cgroup Memory Limit is a Reliability failure seen during ML training runs. Cgroup memory limits cause OOM kills when training exceeds the container or pod's memory limit. Common tags: Cgroup, Memory, Container, Docker.

Container memory limit too low for training workload. Memory leak in training process. Other processes in container consuming memory. GPU memory pinned memory reducing available system memory. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Training crashes with OOM kill
• Process is killed without error
• Container exits with 137 (SIGKILL)

• Docker container training
• Kubernetes pod training
• SLURM jobs with memory limits

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: dmesg: memory cgroup out of memory: Killed process
• ✓Verified signal present: Process killed with SIGKILL (exit code 137)
• ✓Verified signal present: kubelet: Out of memory killing container
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

• Container memory limit too low for training workload
• Memory leak in training process
• Other processes in container consuming memory
• GPU memory pinned memory reducing available system memory