Predictive node health scoring is GA
Per-node 0-100 pre-crash score on the Fleet page and GET /api/fleet/node-health: live ECC/thermal trajectories, Xid/NVLink/PCIe counters and incident recurrence, each deduction named with its evidence. Trend arrows (improving/stable/deteriorating), a recommended action (watch / cordon / drain+RMA), and a failure ETA that only appears when a real monotonic trend supports one, no invented countdowns. Nodes with no telemetry read UNKNOWN, never healthy.
Checkpoint rollback planning + runbooks
POST /api/remediation/rollback-plan builds a verify → cordon (when hardware-implicated) → resume-from-verified-checkpoint plan from a stored diagnosis, with the launcher-specific command detected from the logs. The plan never deletes checkpoints and data-corruption classes stay human-gated. Remote execution remains limited to controlled design-partner pilots until the standard on-prem receiver passes production execute, readback, verification, and rollback qualification.
NCCL Topology Linter, free at /preflight
The deep pre-flight: GPU→NIC affinity from your real nvidia-smi topo -m matrix, PCIe ACS detection (the silent P2P/GDR killer), NCCL_IB_HCA and NCCL_SOCKET_IFNAME validated against your actual HCAs and interfaces, typo detection across 60+ NCCL variables, and memlock ceilings. Runs 100% in your browser, nothing uploaded. Also server-side for CI at POST /api/preflight/topology (Scale+). Corrected 2026-08-12: this entry originally claimed the lint also shipped in the agent. It did not, until the Python port landed on 2026-08-12.
Cross-Run Environment Diff, free at /diff
Paste artifacts from your last good run and the failing one; driver, CUDA, NCCL, PyTorch/JAX, tracked packages, and NCCL_* env are extracted, diffed, and ranked by correlation with the observed failure class, with the weighting shown so the ranking is auditable. Team+ gets the same engine at POST /api/diagnose/env-diff.
Cost optimization advice in every diagnosis
Diagnoses now carry a costAdvisor block: right-sizing backed by measured memory headroom (e.g. your job fits L40S at 60% lower $/GPU-hour), spot↔on-demand reliability trades when a job is crash-looping on spot, and utilization advice, every figure with its basis stated, and a 'measure this first' answer when the evidence isn't there.
GPU Reliability Index, public dataset
The reliability data procurement never gets, at /gpu-reliability: cited field studies (Meta's 419-interruption Llama-3 run on 16,384 H100s; the NCSA Delta A100/H100 resilience study; ORNL's Summit memory-corruption work) alongside the documented failure surface per GPU family, interconnect and provider stack computed from the 559-class public encyclopedia. Machine-readable at GET /api/reliability-index. Deliberately NOT included: a Denpex-fleet MTBF table, that lights up when 90 days of real cross-tenant history exists, not before.
On-call shift handoff reports
GET /api/reports/handoff (and /dashboard/handoff): what failed, what was diagnosed, what auto-remediation did, what the incoming shift inherits, each open incident with its NEXT fix given how far escalation has gone, plus a watchlist (stale running jobs, remediations that need a human). Copy-as-Markdown pastes clean into Slack or a ticket.
Multi-cloud single pane on the Fleet page
Every monitored host across AWS, GCP, Azure, Lambda, CoreWeave, RunPod, OCI and on-prem grouped in one view via GET /api/fleet/multicloud, with per-provider GPU totals, degraded counts, and Denpex-identified open issues. Attribution order: the agent's explicit --provider tag, then hostname conventions; unknown stays its own bucket instead of being folded into on-prem.
CI/CD gates: GitHub Action + GitLab CI template
Gate merges on preflight, the deep topology lint (preflight-deep), version known-bads, or checkpoint validation. Docs at /docs/ci, GitLab template at /ci/denpex-preflight.gitlab-ci.yml. No API key; everything runs on your runner and nothing leaves it. Corrected 2026-08-12: the GitHub example referenced a published Action repository that was never created, and preflight-deep was not implemented in the agent. Both are fixed; the example now calls the single-file agent directly, and a parity test keeps the agent lint identical to the web engine.
Encyclopedia hardening: the 404 wall can't come back
Entry pages now render on demand from static asset copies when the prerender cache misses (dynamicParams=true + a dual-source loader), so the failure mode that once 404'd every /failures/<slug> page now degrades to an edge render instead. Also fixed: every entry title was rendering '… Denpex | Denpex' in search results (double brand), meta descriptions are now answer-first and length-capped, and the live smoke probes cover 11 entry pages across categories including the one dotted slug.
16,384-GPU fleets: mass-crash coalescing
Denpex now supports fleets up to 16,384 GPUs end-to-end. The new mass-crash coalescing engine handles the worst day at that scale: when one fabric event (a bad IB switch, a PDU trip) crashes ~2,000 nodes at once, all their crash reports are grouped by cluster and failure signature, ONE diagnosis runs, and every affected job links to it. One alert with the fix instead of thousands of pages, zero dropped diagnoses, and the whole event bills as a single diagnosis. Crash ingest also moved to a dedicated budget that counts distinct root causes, never raw crash reports. Fleet heartbeats already shard per-tenant with headroom at 2,000 nodes, and cascade analysis is tested to 65,536 ranks. Data Center plan copy updated to match.