Calling a replay a customer success story
The public receipt explicitly marks mocked providers, mocked recipient acknowledgements and the absence of physical recovery or independently certified accuracy.
Controlled replay case study
From a GPU failure alert to an evidence-backed investigation
Download an illustrative local replay of a GPU event, device mapping, notification and recovery checks. It makes no network calls and does not run the hosted Denpex service. A separate internal integration run passed 159 tests using actual application logic with mocked providers and recipient acknowledgements. Neither test failed or repaired a physical GPU.
Denpex monitors GPU clusters and helps engineers investigate failed training and inference jobs. Use this demonstration to inspect the workflow, then evaluate it with one of your own solved incidents and an agreed recovery check.
Reviewed . Reference guidance is not a diagnosis of your workload.
Failure, recommendation and recovery record
- 1
Run the downloadable illustration
Use Node.js 22. Save the linked source as gpu-incident-replay.mjs and inspect it before running. Its assertions cover mapping, duplicate suppression, tenant isolation, missing evidence, contradictory recovery and event order. This independent illustration does not reproduce the private application integration suite or call a model.
node gpu-incident-replay.mjs - 2
Detect events and preserve device identity
The separate internal suite submits authenticated heartbeats into the actual API and MonitorDO logic using SQLite. Its ten-customer scenario represents 960 logical GPUs with 20 fault events. These are simulated devices, not a physical fleet or evidence of 16,384-GPU production capacity. Attribution tests check host, UUID, PCI address and rank boundaries; incomplete identity must remain unresolved.
- 3
Send the initial alert and begin investigation
The internal suite runs the actual formatter and notification outbox against a mocked recipient sink. In the ten-customer scenario, the sink accepted 20 initial notifications and 20 findings. Acceptance by this mock is not inbox delivery, an engineer reading the message or a real provider diagnosis.
- 4
Keep work intact during faults and duplicate events
The internal ten-customer scenario submits 60 duplicate deliveries, injects quota exhaustion and restores coordinator state. It records 21 account attempts for 20 model calls and a peak of four active mocked model requests. A separate capacity-deferral scenario resumes all ten deferred jobs with their retained evidence. These counts validate that replay, not live account throughput.
- 5
Check recovery against explicit evidence
The internal suite exercises recovery-evidence evaluation and contradictory or incomplete evidence boundaries. Physical workload recovery is not established by this replay. In a connected evaluation, retain the approved change, rerun the original workload and compare useful progress, expected outputs and recurrence before reporting observed recovery.
What does this demonstration establish?
| Signal | What it means | Next action |
|---|---|---|
| 159 tests passed with actual SQLite and application logic | The tested software workflow behaved as asserted in the retained run. | Inspect the recorded source hashes and scope. Staff with source access can rerun the internal harness. |
| Mock sink accepted a notification | Formatting, dispatch and acknowledgement logic ran. | Validate real notification delivery separately on your configured channel. |
| A mocked model returned findings | Investigation routing and evidence continuity were exercised. | Assess real first-answer diagnosis quality on a separately labeled incident set. |
| A simulated recovery check passed | The evaluator accepted the supplied test evidence. | Require actual workload results before claiming customer recovery. |
Evidence checklist
- Source commit and receipt hashes
- Original heartbeat and incident identity
- Notification acceptance versus delivery
- First findings and missing evidence
- Post-change workload results
Common mistakes
Tools and deeper references
Open the standalone replay source (Node.js)
Inspect and run the small illustrative fixture. No account, provider call or GPU required.
Inspect the replay receipt (JSON)
Read the tested scope, source hashes, counts and observations.
Inspect a separate RTX 4090 workload replay
Review retained first-answer defects and output checks from a different controlled software incident.
Build the failed-job monitoring checklist
Identify the workload, device evidence and recovery criteria.
Evaluate Denpex on one workload
Start with your own evidence and the current trial terms.
Official references
Frequently asked questions
Did this test use a real 960-GPU cluster?
No. The scenario uses 960 logical device records and injected events. Model responses and recipient acknowledgements are mocked.
Does it prove every crash diagnosis is correct?
No. It validates the software workflow and specified boundaries. Independent incident accuracy, physical recovery and real-provider capacity require separate evidence.
Live diagnosis, no signup
Upload your GPU log. Find what happened first.
Get the likely cause, supporting evidence, and next safe steps with the same cloud reasoning used by paid plans. Allow a few minutes for a thorough analysis. No account, card, or installation. Logs are masked at Denpex API ingress and are not saved to account history.
Before uploading, review cloud data handling and local options.
training_logs.txt
No log to hand? Try one:
Upload your log file. Large files are processed in your browser. Only selected evidence is submitted when you run diagnosis.
3 free diagnoses/day
Try Denpex for 30 days. FREE.
No credit card. No automatic subscription. See whether Denpex helps with an incident you already understand.
Start your free 30-day Team evaluation
Up to 50 diagnoses a day for 30 days. Verify your work email to activate an eligible company evaluation, then replay a resolved incident and compare the evidence and next steps with your team's confirmed cause.
Apply the method to your incident
Use the three free diagnoses to review your error and relevant evidence. Keep reference guidance separate from the cause and recovery status of your own workload.
Diagnose your incident