Skip to content
Evidence and evaluation transparency

What Denpex can prove today.

Diagnostic claims are easy to overstate. This page separates internal development measurements from independent evidence and real customer outcomes, including the gaps that still prevent us from publishing an accuracy percentage.

Download the verified-fix postmortem template (PDF, 6 pages)

Current evidence status

A

Development benchmark

Internal evidence

Labels were hidden during scoring, but the incidents were previously visible during engine development. The result is useful for regression testing, not customer accuracy.

B

Sealed holdout

Not yet published

Zero independently acquired production holdout results are currently published. Until that changes, Denpex does not present an unseen-incident accuracy rate.

C

Customer outcomes

No public cohort

Denpex records authenticated fix feedback and recovery verification for each tenant's operational loop. We do not currently publish aggregate customer outcome statistics.

Live single-GPU workstation capture (RTX 4090)

Real hardware telemetry and crash-wrapper output recorded directly on a physical NVIDIA GeForce RTX 4090 triggering CUDA_ERROR_OUT_OF_MEMORY through nvcuda.dll, processed locally by the canonical offline engine.

Hardware telemetry and preflight

NVIDIA GeForce RTX 4090
PCIe bus and link
00000000:0B:00.0 (Gen 1 active (Gen 4 max) x16)
Driver and VRAM
Driver 621.01 (24564 MiB)
Wrapper ingest captured
1 heartbeat, 1 crash payload
Payload SHA-256
c522cf8a44a0bca26c84e423f81a36ad727941b643820e4eb82910e65c8f69f2
[denpex] GPU preflight
[denpex] 1 NVIDIA GPU(s) found
[denpex] privacy: local checks only. No account. No GPU data uploaded.
GPU 0  NVIDIA GeForce RTX 4090  00000000:0B:00.0  PCIe x16 of x16
[WARN] GPU 0 already holds 2107 MiB of VRAM
RESULT: REVIEW BEFORE LAUNCH (0 blockers, 1 warning in 0.1 seconds)

Canonical engine output

CUDA_OOM

A CUDA allocation failed. The pressure source is unresolved until allocator, workload and competing-process evidence distinguish it.

Recommended action

Capture torch.cuda.memory_summary(abbreviated=False) at the failing allocation and the affected GPU process-memory listing. Change one evidence-supported setting, then replay the failing batch. For immediate capacity relief, reduce per-device microbatch or sequence length and replay the same failing input. This is a controlled test of working-set pressure, not proof of a particular layer. Observed allocation context: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 256.00 GiB. GPU 0 has a total capacity of 23.99 GiB of which 22.50 GiB is free. Including non-PyTorch memory, this process has 1.49 GiB memory in use. CUDA driver error: CUDA_ERROR_OUT_OF_MEMORY (out of memory, rc=2).

Traceback (most recent call last):
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 256.00 GiB. GPU 0 has a total capacity of 23.99 GiB of which 22.50 GiB is free. Including non-PyTorch memory, this process has 1.49 GiB memory in use. CUDA driver error: CUDA_ERROR_OUT_OF_MEMORY (out of memory, rc=2).
# ===== end historical signal hits, begin crash tail =====
[rtx4090-runner] device=NVIDIA GeForce RTX 4090 cuda_driver=13.5 total_gib=23.99 free_gib=22.50 vram_probe_rc=0
Traceback (most recent call last):
  File "train_rtx4090_probe.py", line 42, in <module>
    cuMemAlloc_v2(256.00 GiB) on GPU 0 (NVIDIA GeForce RTX 4090)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 256.00 GiB. GPU 0 has a total capacity of 23.99 GiB of which 22.50 GiB is free. Including non-PyTorch memory, this process has 1.49 GiB memory in use. CUDA driver error: CUDA_ERROR_OUT_OF_MEMORY (out of memory, rc=2).

Reproduction command

python scripts/capture-local-gpu-proof.py

Engine 2026.11.21+trust-provider-cleanup-r1 · captured 2026-11-22T02:19:51Z

Single-GPU scope boundary

Captured locally on an NVIDIA GeForce RTX 4090 (driver 621.01, PCIe 00000000:0B:00.0) running scripts/capture-local-gpu-proof.py against an ephemeral localhost ingest endpoint with zero external transmission. This single-GPU workstation capture proves real host preflight, agent crash shipping, and deterministic classification, and does not constitute multi-node NVLink/InfiniBand validation or a customer cohort.

Public multi-GPU GitHub crash traces

Public closed GitHub crash tracebacks from upstream multi-GPU framework repositories, paired with maintainer or reporter confirmed resolutions. These four public traces show real non-synthetic failure logs processed deterministically, and do not represent a statistically sampled unseen holdout accuracy cohort.

vLLM · vllm-project/vllm#52778CUDA_API_INVALID_ARGUMENT

CustomAllreduce admits cross-node TP=2 groups and dies in cudaIpcOpenMemHandle

Setup: Two NVIDIA GB200 NVL72 nodes, 1 GPU per node, tensor_parallel_size=2 with CUDA graph capture

vllm/executor/ray_utils.py:329  WARNING  tensor_parallel_size=2 exceeds available GPUs per node; TP workers spread across 2+ nodes   (emitted twice)
vllm/distributed/device_communicators/cuda_communicator.py:266  using CUSTOM all-reduce for group tp:0
vllm/v1/worker/gpu_model_runner.py:6816  Profiling CUDA graph memory: PIECEWISE=83, FULL=83
Cuda error /workspace/csrc/custom_all_reduce.cuh:143 'invalid resource handle'

Denpex engine output

CUDA argument/launch-configuration rejection on the affected rank (not identified in the submitted logs), this is the root cause. Stronger lower-layer evidence wins over downstream CUDA/NCCL/framework symptoms; fix this initiating fault first.

Maintainer / issue resolution

Maintainer merged fix PR #55835 alongside FlashInfer all-reduce default (#52998) to prevent invalid cross-node CustomAllreduce IPC handle opens on TP=2 workers split across hosts.

SGLang · sgl-project/sglang#11382CUDA_OOM

Qwen3-VL-235B-A22B-Instruct multi-GPU serving CUDA out of memory

Setup: 140.06 GiB GPU hosting multi-process SGLang weights and CUDA Graph private pools

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 512.00 MiB. GPU 0 has a total capacity of 140.06 GiB of which 424.06 MiB is free. Process 3939595 has 4.48 GiB memory in use. Process 3952261 has 135.16 GiB memory in use. Of the allocated memory 128.30 GiB is allocated by PyTorch, with 360.00 MiB allocated in private pools (e.g., CUDA Graphs), and 2.92 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.

Denpex engine output

A CUDA allocation failed. The pressure source is unresolved until allocator, workload and competing-process evidence distinguish it.

Maintainer / issue resolution

Issue author confirmed that capping static KV-cache reservation with --mem-fraction-static 0.8 eliminated the OOM failure and stabilized startup.

PyTorch Distributed · pytorch/pytorch#143711TORCH_DISTRIBUTED_PORT_CONFLICT

DistNetworkError address already in use during spawned dataloader workers

Setup: PyTorch distributed worker spawn colliding on rendezvous store socket 55733 (errno 98)

Epoch 0
torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address.
The server socket has failed to bind to [::]:55733 (errno: 98 - Address already in use).
The server socket has failed to bind to 0.0.0.0:55733 (errno: 98 - Address already in use).

Denpex engine output

The rendezvous port is already bound on the master host, so the TCPStore cannot listen and the job dies before `init_process_group`. Nothing has reached the GPUs, this is a host port-allocation collision, usually a second job (or an orphaned one) holding the same port.

Maintainer / issue resolution

PyTorch contributor confirmed global-scope Fabric initialization re-executed inside spawned worker processes, attempting to re-bind the same rendezvous port.

DeepSpeed · deepspeedai/DeepSpeed#7242CUDA_OOM

OpenRLHF AutoTP multi-rank SFT training CUDA out of memory on 80 GiB nodes

Setup: Multi-rank distributed training (rank 1 and rank 2 shown) on 79.14 GiB GPUs running DeepSpeed backward pass

[rank1]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.96 GiB. GPU 1 has a total capacity of 79.14 GiB of which 1.01 GiB is free. Including non-PyTorch memory, this process has 0 bytes memory in use. Of the allocated memory 76.49 GiB is allocated by PyTorch, and 76.30 MiB is reserved by PyTorch but unallocated.
[rank2]: Traceback (most recent call last):
[rank2]:   File "/openrlhf/openrlhf/cli/train_sft.py", line 284, in <module>
[rank2]:     train(args)
[rank2]:   File "/openrlhf/openrlhf/trainer/sft_trainer.py", line 154, in fit
[rank2]:     self.strategy.backward(loss, self.model, self.optimizer)
[rank2]:   File "/openrlhf/openrlhf/utils/deepspeed/deepspeed.py", line 142, in backward
[rank2]:     model.backward(loss)

Denpex engine output

A CUDA allocation failed on rank 1. Reported active PyTorch allocations occupy 97 percent of device capacity, and the request exceeds free memory. This supports working-set pressure, but does not identify a particular layer.

Maintainer / issue resolution

DeepSpeed maintainer clarified optimizer_states sharding across dp_groups for ZeRO-1/ZeRO-2 alongside AutoTP and recommended inspecting print(model) to confirm layer partitioning (including unsharded embedding and lm_head weights).

RMA evidence packet

This seeded packet shows the evidence boundary before hardware escalation. One uncorrectable ECC event earns a field-diagnostic verdict, not an automatic replacement request.

Verdict tier

FIELD_DIAG_FIRST

Uncorrectable memory error observed. Capture a bug report before an approved reset, check row-remap status, then run diagnostics on an isolated GPU. The vendor determines RMA eligibility.

Single observed Xid event. Recurrence is not established.

Device identity

vendor
NVIDIA
hostname
node-05
gpu Uuid
GPU-9f8e7d6c-1234-4abc-9def-0123456789ab
gpu Serial
1560921003217
pci Bus Id
0000:8a:00
driver Version
550.90.07
vbios Version
96.00.99.00.01

Missing evidence before submission

  • Missing: System / chassis serial (service tag)
  • Missing: nvidia-bug-report archive (NVIDIA delays RMAs without it)
  • Missing: DCGM diagnostic result
  • Missing: NVIDIA Field Diagnostic result
  • Missing: BMC system event log (server-vendor cases)

Verification controls

  1. nvidia-smi -q -d ROW_REMAPPER # check "Remapping Failure Occurred" and pending remaps
  2. sudo nvidia-bug-report.sh # capture evidence before any reset or power cycle
  3. Manual approval required before draining node-05; isolate workloads before diagnostics.
  4. Manual approval required before resetting GPU 00000000; confirm that no workload owns it.
  5. Manual approval required before dcgmi diag -r 3; run it only on an isolated idle GPU.
  6. Manual approval required before dcgmi diag -r 4; use extended stress only on an isolated idle GPU.
  7. Request NVIDIA Field Diagnostic from the vendor before deciding on RMA eligibility.

Support-ready text

Vendor acceptance is not claimed. This packet organizes support evidence, but the vendor makes the warranty and replacement decision.

GPU HARDWARE REVIEW DRAFT (denpex.rma.v2)
-----------------------------------------------------------
Verdict: FIELD_DIAG_FIRST
Reason: Uncorrectable memory error observed. Capture a bug report before an approved reset, check row-remap status, then run diagnostics on an isolated GPU. The vendor determines RMA eligibility.
Vendor: NVIDIA
Host: node-05   System serial: Not found, run dmidecode
GPU UUID: GPU-9f8e7d6c-1234-4abc-9def-0123456789ab
GPU serial: 1560921003217
PCI bus: 0000:8a:00   Driver: 550.90.07   VBIOS: 96.00.99.00.01
Time of failure: Jun 10 08:14:19
Fault signature: GPU_XID_48   Xid codes: 48
Evidence checklist:
  [x] GPU UUID
  [x] GPU serial number
  [ ] System / chassis serial (service tag), collect via: sudo dmidecode -s system-serial-number
  [x] Driver + VBIOS versions
  [x] Xid excerpt with timestamps
  [x] ECC / row-remap counters
  [ ] nvidia-bug-report archive (NVIDIA delays RMAs without it), collect via: sudo nvidia-bug-report.sh   # run on the failing node, attach nvidia-bug-report.log.gz
  [ ] DCGM diagnostic result, collect via: dcgmi diag -r 3   # run on an isolated idle GPU
  [ ] NVIDIA Field Diagnostic result, collect via: Request the Field Diagnostic procedure and tool from NVIDIA or your hardware vendor
  [ ] BMC system event log (server-vendor cases), collect via: ipmitool sel elist | tail -50
Xid events:
  Jun 10 08:14:19 node-05 kernel: NVRM: Xid (PCI:0000:8a:00): 48, pid=4012, name=python, Double Bit ECC Error
ECC / remap evidence:
  Jun 10 08:14:19 node-05 kernel: NVRM: Xid (PCI:0000:8a:00): 48, pid=4012, name=python, Double Bit ECC Error
  [rank42] ECC uncorrectable error detected on GPU 00000000:8A:00.0
This is an evidence draft, not an RMA approval. Run the verification flow and ask the vendor to determine eligibility.
Generated by Denpex ML Incident Intelligence.

Evidence provenance

Seeded public demonstration derived from committed RMA regression cases. It contains no customer or vendor case data.

Tenant memory and verified recovery

This seeded replay shows how a team's authenticated prior fix is replayed alongside the engine recommendation, attributed and advisory rather than overwriting it, while recovery remains open until the required server-clock observation window finishes.

Prior confirmed fix replay

Confirmed 3 times

Your team's confirmed fix for NCCL_TIMEOUT (last confirmed 2026-08-20 by seeded-operator@example.invalid, confirmed 3 times):

Pin NCCL_IB_HCA to mlx5_0 and rerun the same collective control.

Engine recommendation

Engine recommendation: inspect the first transport error and compare a known-good rail.

Fix source
team-kb (advisory)
Last confirmed
2026-08-20

Verification report

Awaiting observation

Recovery remains open. A passing no-recurrence report is still pending until its full 30-minute window elapses.

  • The GPU enumerates again through NVMLpassed

    Required. Observation window: 0 minutes.

  • The same incident did not recurpending

    Required. Observation window: 30 minutes.

Tenant isolation

This confirmed fix belongs to seeded Team Alpha only. It is not public knowledge and is not available to another tenant.

Evidence provenance

Seeded public demonstration of production team-memory and verification contracts. It contains no customer data.

First-rank localization with telemetry

In this seeded fixture, rank 3 reports the first primary fault. Ranks 0, 1, 2 report later collective timeouts and are collateral victims. The telemetry overlay records one volatile uncorrectable ECC event on the initiating GPU.

One GPU fault, three collateral rank timeouts
TimeRank and nodeCausal roleEvidence
12:00:00.120 UTCRank 3, node-bInitiator[rank3] CUDA error: uncorrectable ECC error encountered
12:00:02.100 UTCRank 0, node-aCollateral victim[rank0] Watchdog caught collective operation timeout
12:00:02.220 UTCRank 1, node-aCollateral victim[rank1] NCCL timeout waiting for all-reduce
12:00:02.310 UTCRank 2, node-aCollateral victim[rank2] collective operation timeout

Evidence provenance

Seeded public demonstration derived from committed deterministic regression cases. It is not a customer incident or an accuracy sample.

Uncertainty boundary

This fixture demonstrates one covered causal pattern. It does not establish correctness for incomplete incidents, other failure families, or production populations.

Development evidence, with its limitation attached

What was measured

A reproducible comparison scores the Denpex causal pipeline and three specified algorithmic heuristics on the same development corpus. It is useful for detecting regressions and checking whether causal ranking improves on simple signature selection.

What it does not establish

It is not a sealed holdout, a production accuracy estimate, a comparison with ML infrastructure engineers, or a comparison with general AI assistants. We therefore do not promote its class-match percentage as customer proof.

Reproduction command: npx tsx worker/eval/run-blinded-comparison.ts

The bar for publishable accuracy evidence

Denpex will not call an evaluation independent merely because labels were hidden during one run. A publishable sealed holdout must satisfy every requirement below.

  • Incidents the diagnostic engine and its authors have never used for tuning
  • Maintainer-confirmed or operator-confirmed root causes and resolutions
  • Semantic grading of the diagnosis, evidence, commands, verification, and abstention
  • Coverage across failure families, frameworks, schedulers, accelerators, and incomplete evidence
  • A frozen evaluation protocol and published sample counts, limitations, and retired cases

Customer-observed outcomes

No consented customer outcome sample is available for public reporting yet. A future public result must separate human-confirmed outcomes from automated checks, suppress small cohorts, prevent duplicate submissions, and honor deletion and consent boundaries.

Until those controls and a sufficient independent cohort are production-verified, this page will remain an explicit empty state instead of displaying zeroes as if they were evidence.

Inspect the product directly

Run the public matcher or review the architecture and safety boundaries behind the diagnosis pipeline.