Using the full job duration as lost compute
Count the work that must be repeated. Progress preserved by a restorable checkpoint remains available.
Operator task guide
How much does a failed GPU training job cost?
Measure work lost since the last usable checkpoint, paid idle capacity before restart, and active engineering time. Use scheduler timestamps and your effective GPU rate for each interval.
Keep these amounts separate so the same GPU hours are counted once. Treat a shared reservation as a capacity cost unless the outage caused an incremental charge.
Step-by-step method
- 1
Find the last usable checkpoint
Use the most recent checkpoint that the workload can actually restore. Compare its timestamp or step with the failure and count the GPU hours needed to redo that work.
- 2
Count paid idle capacity
Use allocation and billing records to determine how many GPUs stayed reserved while the team investigated. Track queue delay separately when no capacity was billed.
- 3
Calculate an example
For 64 GPUs at an assumed rate of $3 per GPU-hour, 30 minutes of lost work costs $96. Another 15 minutes of billed idle allocation costs $48. At an assumed loaded rate of $120 per hour, 45 minutes of active investigation costs $90. The total is $234 under these assumptions.
- 4
Measure what changes in an evaluation
Record alert time, time to a supported next action, and time to verified restart on a comparable incident. Use actual allocation records to calculate any observed cost difference.
Which costs belong in the estimate?
| Signal | What it means | Next action |
|---|---|---|
| Training progress must be replayed | There is measurable rework from the last usable checkpoint. | Multiply repeated GPU hours by the effective rate. |
| GPUs stayed billed while idle | Reserved capacity carries a cost during the outage. | Use the billed idle interval and exclude any overlapping work. |
| An engineer investigated the crash | Active labor is a separate cost. | Multiply active investigation time by the loaded hourly rate. |
Evidence checklist
- GPU count and effective rate
- Last usable checkpoint and failure time
- Billed idle interval
- Active investigation time
- Time when useful work resumed
Common mistakes
Frequently asked questions
Does time waiting in the queue count as GPU spend?
Count queue time as billed GPU spend only when the corresponding capacity remained allocated and billed. You can report time lost separately.
Live diagnosis, no signup
Upload your GPU log. Find what happened first.
Get the likely cause, supporting evidence, and next safe steps with the same cloud reasoning used by paid plans. Allow a few minutes for a thorough analysis. No account, card, or installation. Logs are masked at Denpex API ingress and are not saved to account history.
Before uploading, review cloud data handling and local options.
training_logs.txt
No log to hand? Try one:
Upload your log file. Large files are processed in your browser. Only selected evidence is submitted when you run diagnosis.
3 free diagnoses/day
Try Denpex for 30 days. FREE.
No credit card. No automatic subscription. See whether Denpex helps with an incident you already understand.
Start your free 30-day Team evaluation
Up to 50 diagnoses a day for 30 days. Verify your work email to activate an eligible company evaluation, then replay a resolved incident and compare the evidence and next steps with your team's confirmed cause.
Apply the method to your incident
Use the three free diagnoses to review your error and relevant evidence. Keep reference guidance separate from the cause and recovery status of your own workload.
Diagnose your incident