Development benchmark
Internal evidence
Labels were hidden during scoring, but the incidents were previously visible during engine development. The result is useful for regression testing, not customer accuracy.
Diagnostic claims are easy to overstate. This page separates internal development measurements from independent evidence and real customer outcomes, including the gaps that still prevent us from publishing an accuracy percentage.
Download the verified-fix postmortem template (PDF, 6 pages)Internal evidence
Labels were hidden during scoring, but the incidents were previously visible during engine development. The result is useful for regression testing, not customer accuracy.
Not yet published
Zero independently acquired production holdout results are currently published. Until that changes, Denpex does not present an unseen-incident accuracy rate.
No public cohort
Denpex records authenticated fix feedback and recovery verification for each tenant's operational loop. We do not currently publish aggregate customer outcome statistics.
Real hardware telemetry and crash-wrapper output recorded directly on a physical NVIDIA GeForce RTX 4090 triggering CUDA_ERROR_OUT_OF_MEMORY through nvcuda.dll, processed locally by the canonical offline engine.
[denpex] GPU preflight [denpex] 1 NVIDIA GPU(s) found [denpex] privacy: local checks only. No account. No GPU data uploaded. GPU 0 NVIDIA GeForce RTX 4090 00000000:0B:00.0 PCIe x16 of x16 [WARN] GPU 0 already holds 2107 MiB of VRAM RESULT: REVIEW BEFORE LAUNCH (0 blockers, 1 warning in 0.1 seconds)
A CUDA allocation failed. The pressure source is unresolved until allocator, workload and competing-process evidence distinguish it.
Recommended action
Capture torch.cuda.memory_summary(abbreviated=False) at the failing allocation and the affected GPU process-memory listing. Change one evidence-supported setting, then replay the failing batch. For immediate capacity relief, reduce per-device microbatch or sequence length and replay the same failing input. This is a controlled test of working-set pressure, not proof of a particular layer. Observed allocation context: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 256.00 GiB. GPU 0 has a total capacity of 23.99 GiB of which 22.50 GiB is free. Including non-PyTorch memory, this process has 1.49 GiB memory in use. CUDA driver error: CUDA_ERROR_OUT_OF_MEMORY (out of memory, rc=2).
Traceback (most recent call last):
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 256.00 GiB. GPU 0 has a total capacity of 23.99 GiB of which 22.50 GiB is free. Including non-PyTorch memory, this process has 1.49 GiB memory in use. CUDA driver error: CUDA_ERROR_OUT_OF_MEMORY (out of memory, rc=2).
# ===== end historical signal hits, begin crash tail =====
[rtx4090-runner] device=NVIDIA GeForce RTX 4090 cuda_driver=13.5 total_gib=23.99 free_gib=22.50 vram_probe_rc=0
Traceback (most recent call last):
File "train_rtx4090_probe.py", line 42, in <module>
cuMemAlloc_v2(256.00 GiB) on GPU 0 (NVIDIA GeForce RTX 4090)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 256.00 GiB. GPU 0 has a total capacity of 23.99 GiB of which 22.50 GiB is free. Including non-PyTorch memory, this process has 1.49 GiB memory in use. CUDA driver error: CUDA_ERROR_OUT_OF_MEMORY (out of memory, rc=2).python scripts/capture-local-gpu-proof.py
Engine 2026.11.21+trust-provider-cleanup-r1 · captured 2026-11-22T02:19:51Z
Captured locally on an NVIDIA GeForce RTX 4090 (driver 621.01, PCIe 00000000:0B:00.0) running scripts/capture-local-gpu-proof.py against an ephemeral localhost ingest endpoint with zero external transmission. This single-GPU workstation capture proves real host preflight, agent crash shipping, and deterministic classification, and does not constitute multi-node NVLink/InfiniBand validation or a customer cohort.
Public closed GitHub crash tracebacks from upstream multi-GPU framework repositories, paired with maintainer or reporter confirmed resolutions. These four public traces show real non-synthetic failure logs processed deterministically, and do not represent a statistically sampled unseen holdout accuracy cohort.
Setup: Two NVIDIA GB200 NVL72 nodes, 1 GPU per node, tensor_parallel_size=2 with CUDA graph capture
vllm/executor/ray_utils.py:329 WARNING tensor_parallel_size=2 exceeds available GPUs per node; TP workers spread across 2+ nodes (emitted twice) vllm/distributed/device_communicators/cuda_communicator.py:266 using CUSTOM all-reduce for group tp:0 vllm/v1/worker/gpu_model_runner.py:6816 Profiling CUDA graph memory: PIECEWISE=83, FULL=83 Cuda error /workspace/csrc/custom_all_reduce.cuh:143 'invalid resource handle'
Denpex engine output
CUDA argument/launch-configuration rejection on the affected rank (not identified in the submitted logs), this is the root cause. Stronger lower-layer evidence wins over downstream CUDA/NCCL/framework symptoms; fix this initiating fault first.
Maintainer / issue resolution
Maintainer merged fix PR #55835 alongside FlashInfer all-reduce default (#52998) to prevent invalid cross-node CustomAllreduce IPC handle opens on TP=2 workers split across hosts.
Setup: 140.06 GiB GPU hosting multi-process SGLang weights and CUDA Graph private pools
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 512.00 MiB. GPU 0 has a total capacity of 140.06 GiB of which 424.06 MiB is free. Process 3939595 has 4.48 GiB memory in use. Process 3952261 has 135.16 GiB memory in use. Of the allocated memory 128.30 GiB is allocated by PyTorch, with 360.00 MiB allocated in private pools (e.g., CUDA Graphs), and 2.92 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.
Denpex engine output
A CUDA allocation failed. The pressure source is unresolved until allocator, workload and competing-process evidence distinguish it.
Maintainer / issue resolution
Issue author confirmed that capping static KV-cache reservation with --mem-fraction-static 0.8 eliminated the OOM failure and stabilized startup.
Setup: PyTorch distributed worker spawn colliding on rendezvous store socket 55733 (errno 98)
Epoch 0 torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. The server socket has failed to bind to [::]:55733 (errno: 98 - Address already in use). The server socket has failed to bind to 0.0.0.0:55733 (errno: 98 - Address already in use).
Denpex engine output
The rendezvous port is already bound on the master host, so the TCPStore cannot listen and the job dies before `init_process_group`. Nothing has reached the GPUs, this is a host port-allocation collision, usually a second job (or an orphaned one) holding the same port.
Maintainer / issue resolution
PyTorch contributor confirmed global-scope Fabric initialization re-executed inside spawned worker processes, attempting to re-bind the same rendezvous port.
Setup: Multi-rank distributed training (rank 1 and rank 2 shown) on 79.14 GiB GPUs running DeepSpeed backward pass
[rank1]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.96 GiB. GPU 1 has a total capacity of 79.14 GiB of which 1.01 GiB is free. Including non-PyTorch memory, this process has 0 bytes memory in use. Of the allocated memory 76.49 GiB is allocated by PyTorch, and 76.30 MiB is reserved by PyTorch but unallocated. [rank2]: Traceback (most recent call last): [rank2]: File "/openrlhf/openrlhf/cli/train_sft.py", line 284, in <module> [rank2]: train(args) [rank2]: File "/openrlhf/openrlhf/trainer/sft_trainer.py", line 154, in fit [rank2]: self.strategy.backward(loss, self.model, self.optimizer) [rank2]: File "/openrlhf/openrlhf/utils/deepspeed/deepspeed.py", line 142, in backward [rank2]: model.backward(loss)
Denpex engine output
A CUDA allocation failed on rank 1. Reported active PyTorch allocations occupy 97 percent of device capacity, and the request exceeds free memory. This supports working-set pressure, but does not identify a particular layer.
Maintainer / issue resolution
DeepSpeed maintainer clarified optimizer_states sharding across dp_groups for ZeRO-1/ZeRO-2 alongside AutoTP and recommended inspecting print(model) to confirm layer partitioning (including unsharded embedding and lm_head weights).
This seeded packet shows the evidence boundary before hardware escalation. One uncorrectable ECC event earns a field-diagnostic verdict, not an automatic replacement request.
Uncorrectable memory error observed. Capture a bug report before an approved reset, check row-remap status, then run diagnostics on an isolated GPU. The vendor determines RMA eligibility.
Single observed Xid event. Recurrence is not established.
Vendor acceptance is not claimed. This packet organizes support evidence, but the vendor makes the warranty and replacement decision.
GPU HARDWARE REVIEW DRAFT (denpex.rma.v2) ----------------------------------------------------------- Verdict: FIELD_DIAG_FIRST Reason: Uncorrectable memory error observed. Capture a bug report before an approved reset, check row-remap status, then run diagnostics on an isolated GPU. The vendor determines RMA eligibility. Vendor: NVIDIA Host: node-05 System serial: Not found, run dmidecode GPU UUID: GPU-9f8e7d6c-1234-4abc-9def-0123456789ab GPU serial: 1560921003217 PCI bus: 0000:8a:00 Driver: 550.90.07 VBIOS: 96.00.99.00.01 Time of failure: Jun 10 08:14:19 Fault signature: GPU_XID_48 Xid codes: 48 Evidence checklist: [x] GPU UUID [x] GPU serial number [ ] System / chassis serial (service tag), collect via: sudo dmidecode -s system-serial-number [x] Driver + VBIOS versions [x] Xid excerpt with timestamps [x] ECC / row-remap counters [ ] nvidia-bug-report archive (NVIDIA delays RMAs without it), collect via: sudo nvidia-bug-report.sh # run on the failing node, attach nvidia-bug-report.log.gz [ ] DCGM diagnostic result, collect via: dcgmi diag -r 3 # run on an isolated idle GPU [ ] NVIDIA Field Diagnostic result, collect via: Request the Field Diagnostic procedure and tool from NVIDIA or your hardware vendor [ ] BMC system event log (server-vendor cases), collect via: ipmitool sel elist | tail -50 Xid events: Jun 10 08:14:19 node-05 kernel: NVRM: Xid (PCI:0000:8a:00): 48, pid=4012, name=python, Double Bit ECC Error ECC / remap evidence: Jun 10 08:14:19 node-05 kernel: NVRM: Xid (PCI:0000:8a:00): 48, pid=4012, name=python, Double Bit ECC Error [rank42] ECC uncorrectable error detected on GPU 00000000:8A:00.0 This is an evidence draft, not an RMA approval. Run the verification flow and ask the vendor to determine eligibility. Generated by Denpex ML Incident Intelligence.
Seeded public demonstration derived from committed RMA regression cases. It contains no customer or vendor case data.
This seeded replay shows how a team's authenticated prior fix is replayed alongside the engine recommendation, attributed and advisory rather than overwriting it, while recovery remains open until the required server-clock observation window finishes.
Your team's confirmed fix for NCCL_TIMEOUT (last confirmed 2026-08-20 by seeded-operator@example.invalid, confirmed 3 times):
Pin NCCL_IB_HCA to mlx5_0 and rerun the same collective control.
Engine recommendation
Engine recommendation: inspect the first transport error and compare a known-good rail.
Recovery remains open. A passing no-recurrence report is still pending until its full 30-minute window elapses.
Required. Observation window: 0 minutes.
Required. Observation window: 30 minutes.
This confirmed fix belongs to seeded Team Alpha only. It is not public knowledge and is not available to another tenant.
Seeded public demonstration of production team-memory and verification contracts. It contains no customer data.
In this seeded fixture, rank 3 reports the first primary fault. Ranks 0, 1, 2 report later collective timeouts and are collateral victims. The telemetry overlay records one volatile uncorrectable ECC event on the initiating GPU.
| Time | Rank and node | Causal role | Evidence |
|---|---|---|---|
| 12:00:00.120 UTC | Rank 3, node-b | Initiator | [rank3] CUDA error: uncorrectable ECC error encountered |
| 12:00:02.100 UTC | Rank 0, node-a | Collateral victim | [rank0] Watchdog caught collective operation timeout |
| 12:00:02.220 UTC | Rank 1, node-a | Collateral victim | [rank1] NCCL timeout waiting for all-reduce |
| 12:00:02.310 UTC | Rank 2, node-a | Collateral victim | [rank2] collective operation timeout |
Seeded public demonstration derived from committed deterministic regression cases. It is not a customer incident or an accuracy sample.
This fixture demonstrates one covered causal pattern. It does not establish correctness for incomplete incidents, other failure families, or production populations.
A reproducible comparison scores the Denpex causal pipeline and three specified algorithmic heuristics on the same development corpus. It is useful for detecting regressions and checking whether causal ranking improves on simple signature selection.
It is not a sealed holdout, a production accuracy estimate, a comparison with ML infrastructure engineers, or a comparison with general AI assistants. We therefore do not promote its class-match percentage as customer proof.
Reproduction command: npx tsx worker/eval/run-blinded-comparison.ts
Denpex will not call an evaluation independent merely because labels were hidden during one run. A publishable sealed holdout must satisfy every requirement below.
No consented customer outcome sample is available for public reporting yet. A future public result must separate human-confirmed outcomes from automated checks, suppress small cohorts, prevent duplicate submissions, and honor deletion and consent boundaries.
Until those controls and a sufficient independent cohort are production-verified, this page will remain an explicit empty state instead of displaying zeroes as if they were evidence.
Run the public matcher or review the architecture and safety boundaries behind the diagnosis pipeline.