Treating a restart as recovery
Record whether the original workload resumed useful progress and whether the failure returned during observation.
Operator task guide
GPU monitoring for failed training jobs
Start with a job failure alert that identifies the workload, attempt and time window. Attach scheduler state, per-rank logs and host events so the engineer can investigate the likely cause before restarting.
Evaluate the workflow on one workload. Confirm the right team receives the alert, can trace the diagnosis to its evidence and can check that the workload actually recovered.
Step-by-step method
- 1
Identify the failed job
Capture the job ID, attempt, first failed step, scheduler exit state and timestamps. Keep attempt IDs distinct so a retry does not overwrite evidence from the original failure.
- 2
Map the failure to the workload and device
Correlate global rank, host, GPU UUID and PCI address within the failure window. Preserve uncertainty when the mapping is missing or the clocks cannot be aligned.
- 3
Walk through a simulated alert
In this illustrative timeline, rank 3 reports a CUDA allocation failure at 02:14:08 UTC. Peer ranks report NCCL timeouts at 02:15:08 UTC. The allocation failure is the first supported symptom. The operator still needs the allocation size and memory state to distinguish capacity pressure from competing usage.
- 4
Verify recovery on the original workload
After the approved correction, rerun a representative workload and check progress, valid outputs and recurrence. Record when the alert arrived, when an engineer had a supported next action and when useful work resumed.
Evidence to attach to the alert
| Signal | What it means | Next action |
|---|---|---|
| Job failed and the rank emitted an exception | The workload logs contain an initiating candidate. | Preserve the first traceback and compare timestamps across ranks. |
| Only surviving ranks emitted timeout messages | The initiating error may be outside the available logs. | Collect scheduler, container and host events for the missing rank. |
| A host GPU event overlaps the failure | Device attribution requires a workload-to-GPU mapping. | Confirm host, UUID, PCI address and timestamp before escalating. |
Evidence checklist
- Job and attempt ID
- First worker error and peer logs
- Rank, host and GPU identity
- Alert delivery timestamp
- Recovery criteria and workload readback
Common mistakes
Frequently asked questions
What should a GPU job failure alert contain?
Include the workload and attempt IDs, failure time, affected host or rank when supported, and a link to the evidence needed for investigation.
Live diagnosis, no signup
Upload your GPU log. Find what happened first.
Get the likely cause, supporting evidence, and next safe steps with the same cloud reasoning used by paid plans. Allow a few minutes for a thorough analysis. No account, card, or installation. Logs are masked at Denpex API ingress and are not saved to account history.
Before uploading, review cloud data handling and local options.
training_logs.txt
No log to hand? Try one:
Upload your log file. Large files are processed in your browser. Only selected evidence is submitted when you run diagnosis.
3 free diagnoses/day
Try Denpex for 30 days. FREE.
No credit card. No automatic subscription. See whether Denpex helps with an incident you already understand.
Start your free 30-day Team evaluation
Up to 50 diagnoses a day for 30 days. Verify your work email to activate an eligible company evaluation, then replay a resolved incident and compare the evidence and next steps with your team's confirmed cause.
Apply the method to your incident
Use the three free diagnoses to review your error and relevant evidence. Keep reference guidance separate from the cause and recovery status of your own workload.
Diagnose your incident