Public roadmap

Pattern corpus over 16,400+ failure signatures with AI fallback for novel errors.

Single Python file, stdlib only, no root, no kernel module.

All alert channels live with severity mapping and quiet hours.

Confirming a fix promotes it into a knowledge base owned by your TEAM, keyed by failure class. Every later diagnosis of that class, for any teammate, leads with the confirmed fix and keeps the engine recommendation underneath. Re-confirming strengthens the entry rather than duplicating it, a corrected fix beats the engine's wording, and nothing is visible outside your organisation. Readable at /knowledge-base.

Public reference for every failure class Denpex diagnoses.

SOC 2 status, sub-processors, data flow, shared-responsibility matrix.

6-page PDF + sales follow-up, with NDA option.

Continue with GitHub on the login page.

Isolates the rank that stalled the collective when NCCL hangs.

Flags known-bad PyTorch, CUDA, cuDNN, and NCCL combinations.

Verifies the framework, toolkit, and library versions are a supported set.

Maps each NVIDIA Xid class to ISOLATE / RESET / REBOOT / RMA, returns the exact operator plan, and supports approval-gated actions through explicitly connected Kubernetes or SLURM control planes. Per-tenant policy, action ceilings, blast-radius guards and readback checks apply; the standard monitoring agent never executes commands.

denpex preflight-cluster: rendezvous, fabric ring-bandwidth probe, IB port sanity, GPU/driver uniformity, ulimits and clock skew. denpex preflight run adds the executable benchmark: real NCCL all-reduce / all-gather / reduce-scatter busbw across a size sweep, HBM and PCIe bandwidth per GPU, and per-rank timing spread that names the straggler before the job starts. Free and account-less, curl -fsSL https://denpex.com/preflight.sh | sh. Exits nonzero on a blocker so it can gate a launch.

Anonymized, aggregated failure signatures pooled across fleets so you're warned about a failure class before it reaches your cluster. Gated on a published cross-tenant data-handling disclosure before any signal is shared.

Approval-gated connected-cluster actions for node cordon/drain, pod eviction, and job requeue. Ray failures are diagnosed, but a Ray control-plane action connector is not yet shipped.

Per-node 0-100 pre-crash score from live ECC/thermal trajectories, Xid/NVLink counters and incident recurrence, with named signals, trend, recommended action (watch/cordon/drain+RMA), and a failure-ETA only when a real trend supports one. Each node also carries a modelled failure probability: a Weibull baseline hazard with the shape fitted by maximum likelihood from your own failures and still-running nodes, adjusted by Welford z-scores against each node's own counter history, shown as a risk meter with a confidence interval. It reports 'prior only' rather than a fitted number until your fleet has enough observed failures, and no number at all when a node's operating hours are unknown. On the Fleet page and GET /api/fleet/node-health.

GPU→NIC affinity, PCIe ACS detection, NCCL_IB_HCA / NCCL_SOCKET_IFNAME validated against the real fabric, memlock limits. Free in-browser at /preflight, in the agent as preflight --deep (offline, keyless, parity-tested against the web engine), and as a Scale+ API for CI gates.

You paste artifacts from the last good run and the failing one; every change (driver, CUDA, NCCL, framework, packages, NCCL_* env) is ranked by correlation with the observed failure class. Free at /diff; Team+ API. This manual path remains useful for historical runs that predate the current agent.

The current agent records a sanitized launch fingerprint, stores successful runs as tenant-scoped baselines, and attaches a diff against the closest comparable successful run of the same job when a crash arrives. If the job has never completed successfully, Denpex says no baseline exists instead of comparing an unrelated run.

Right-sizing with measured-memory evidence, spot↔on-demand reliability trades, and utilization advice, attached to every diagnosis with the basis stated.

Public procurement dataset at /gpu-reliability: cited field studies (Meta's Llama-3 run, the NCSA Delta A100/H100 study) plus the documented failure surface per GPU family, interconnect and provider stack.

The overnight story in one page: what failed, what was diagnosed, what auto-remediation did, what the incoming shift inherits. Dashboard page + copy-as-Markdown + API.

Every monitored host across AWS, GCP, Azure, Lambda, CoreWeave, RunPod and on-prem in one fleet view, with Denpex-identified open issues attributed per provider.

Block the merge that would have burned the GPU-hours: preflight, deep topology lint, version known-bads and checkpoint validation as pipeline gates. No API key required.

Bring your own KMS keys for AES-256 at rest on Data Center.

Published and installable: pip install denpex-sdk · npm install @denpex/sdk · npx denpex-mcp for the Model Context Protocol server. All three call the same diagnosis engine as the console, and the MCP server's cascade and encyclopedia lookup work offline without an account.

Evidence that cannot be reconstructed after the fact has to be captured before the job starts. The agent takes a sanitized run fingerprint at launch, scheduler placement, rank-to-GPU-UUID identity, container image digest, source revision, driver, CUDA, NCCL and framework versions, launcher configuration, dataset shard and checkpoint identity, and assembles a failure bundle at termination: first causal error with preceding events, per-rank logs from the relevant window, Xid, ECC, PCIe, NVLink and fabric evidence, DCGM health and job statistics, NCCL RAS state, scheduler events, exit code with signal and termination reason, and checkpoint integrity. DCGM job statistics in particular are bounded by job start and stop, so they do not exist unless something enabled them beforehand. Environment variables are captured through a deny-by-default sanitizer; secrets are never collected. As of 2026-08-21 the bundle also GROUNDS the diagnosis rather than only being stored beside it: the rank-to-GPU identity, the NCCL RAS unresponsive-rank list, the DCGM counters and the checkpoint state are supplied to the engine as measured fact, each stated with what it does and does not license, a low-confidence rank-to-GPU mapping is reported as probable rather than identified, and counters reading zero are treated as evidence against a hardware cause rather than ignored. It remains in progress because the bundle is only as complete as the node allows: DCGM job statistics need enabling before the workload, and NCCL RAS needs NCCL 2.24 or newer, so what a given cluster can capture varies and the agent reports the gaps rather than papering over them.

The control plane can classify, dry-run, authorize, audit, verify, and roll back governed actions, but the standard public monitoring agent does not yet receive command callbacks. Production qualification requires the on-prem receiver, scheduler-scoped credentials, signed callback replay protection, execution reporting, readback verification, and rollback to pass end-to-end tests. Until then, customers receive exact human-run steps and remote execution remains fail-closed.

Checkpoint integrity analysis and launcher-specific runbooks are available. Hands-off resume stays disabled until the production receiver proves the full verify → execute → progress-readback → rollback loop against real Kubernetes and SLURM test clusters.

JAX/Flax and XLA stack traces already diagnose today via the console's AI fallback. Deterministic packs for pjit/sharding mismatches, XLA OOM, TPU runtime faults, and host-callback deadlocks are under active build, university and national-lab JAX stacks are the design target.

Stream billing, team, and admin events to your SIEM. Exporter is built; hardening the connector and finishing customer-facing config.

Just-in-time provisioning and deprovisioning from your IdP. Endpoint is live; completing IdP conformance and directory-sync wiring.

Okta, Azure AD, Google, OneLogin, custom OIDC via managed AuthKit.

Point-in-time tokens/sec and MFU are live in the Goodput dashboard. Over-time regression detection, slow pipeline-stage localization, and fleet-wide straggler ranking are under active build.

D1 jurisdiction pinning is live; running the entire control plane in eu-west-1 with EU-only data residency is next.

Controls implemented; independent audit planned; report will be available under NDA on completion.