Missing Signal Handler

Missing signal handlers prevent graceful shutdown of training, causing checkpoint loss and resource leaks.

Missing Signal Handler is a Reliability failure seen during ML training runs. Missing signal handlers prevent graceful shutdown of training, causing checkpoint loss and resource leaks. Common tags: Signal, Handler, Graceful Shutdown, Reliability.

Signal handler not registered. Cleanup logic in main thread only. CUDA context not released on signal. Multiple processes not handling signals. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Training doesn't save checkpoint on Ctrl+C
• GPU memory not released on interrupt
• Process killed mid-training without cleanup

• Long-running training
• Production training jobs
• Training that needs graceful shutdown

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: SIGTERM/SIGINT received but training continues
• ✓Verified signal present: CUDA OOM after interrupted training
• ✓Verified signal present: Checkpoints corrupted when process killed
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

• Signal handler not registered
• Cleanup logic in main thread only
• CUDA context not released on signal
• Multiple processes not handling signals