A frontier LLM lab running 70B parameter fine-tunes across a 256-GPU cluster replaced a custom Slack-bot + grep pipeline with Denpex. Here's what changed.
Median time-to-root-cause (was 4 hours)
GPU-hours wasted to misdiagnosed failures
Successful training runs per quarter
The team was running 70B fine-tunes on a 256-GPU H100 cluster under tight deadlines. When a job died, the on-call engineer would spend an average of 4 hours going through logs from 32 nodes to figure out which rank actually failed, then another 30 minutes explaining it to the ML team. Most of that time was the same failures over and over. NCCL timeouts, CUDA OOM, checkpoint corruption. The Slack bot couldn't classify any of them.
The expensive ones were the quiet failures. A dying GPU would bring down the whole job, and nobody knew which node it was until someone SSH'd in and checked nvidia-smi.
Denpex replaced the custom Slack bot. When a job crashes, the diagnosis shows up in Slack in 12 seconds: which rank failed, what the actual error was, and what to run next.
The silent failures got easier too. Denpex catches a degrading GPU within a single training step instead of after it takes the job down. The team ditched the dmesg-watching script they'd been babysitting for two years.
Median time-to-diagnosis went from 4 hours to 12 seconds. GPU waste from misdiagnosed failures dropped 35% in the first quarter. The biggest surprise was the silent hardware failures. They used to take down a job every few weeks and nobody knew why until after. The team thinks they avoided at least two full-cluster restarts in the first month.
The platform team stopped maintaining the regex list entirely. They spend that time on actual reliability work now.
We can connect you with this team directly, under NDA. Or just try it: 3 free diagnoses, no card required.