Install the agent

Install & verify

Python 3.8+. No third-party packages, no kernel module, no root, no daemon. The agent never modifies the training process and never installs anything. It's a single readable file. Verify the checksum, read the source, then run it.

Wrap a training command

Diagnose one saved log

The standalone diagnose command uses the same cloud engine as the homepage, console, API, and MCP server. Without a key it uses the three-per-day test drive; with a key it also records history and applies your plan entitlements.

Environment variables

Make the next failure conclusive

Some evidence cannot be recovered after a job dies. DCGM job statistics are accumulated between an explicit start and stop marker, so a run that finished without them simply has none, there is no later command that reconstructs the power draw, clock-throttle time or per-GPU Xid tally over the window the job occupied. The PyTorch Flight Recorder is the same: it records into a ring buffer that has to be enabled before init_process_group.

Every diagnosis tells you which of these were missing and what to set, so you do not have to remember this page. Setting them ahead of time is what turns “we think it was rank 42” into “it was rank 42, on this serial number, and here is the power trace.”

DCGM job statistics need nothing from you if you run training through denpex.py run, the agent marks the job's start and stop itself. If you launch some other way, call denpex_dcgm.start_job_stats() in your scheduler prolog.

Verify a fix actually worked

A repair command exiting 0 means the command was accepted, not that the node recovered. nvidia-smi -r returning 0 does not mean the GPU came back, and handing that node to the next job is how one fault becomes two. denpex verify reads back the actual state instead: GPU enumeration, new Xids, ECC movement, fabric links, scheduler admission, finite loss, throughput against the last healthy run, and whether the failure stayed away.

It reports one of six outcomes, and only one of them exits 0. These are the same names the control plane uses, so an agent verdict and a dashboard verdict always mean the same thing.

Observation windows are per check and are not settable from the command line, “no fatal Xid” means nothing fifteen seconds after a reset, and an operator should not be able to shorten it into a verdict that means nothing. A check that cannot be performed reports as unavailable, and one that is merely too early reports as pending. Neither is ever counted as a pass.

Kubernetes

Run the read-only node monitor as a DaemonSet. Create the API-key Secret yourself first; the published manifest deliberately contains no placeholder Secret that could overwrite a real credential.

Self-hosted / air-gapped (Helm)

For data centers that can't let telemetry leave their network, run the whole platform on your own hardware with one command. The agent diagnoses failures on-host and serves fleet health as Prometheus metrics, your own Prometheus + Grafana watch the cluster, with zero egress and no per-request cost. Ships as a DaemonSet (one pod per GPU node) from the Denpex on-prem kit.

The chart ships a pinned, scripts-baked-in container image. No repo checkout, no manual ConfigMap. A runnable starter stack (Prometheus + VictoriaMetrics + Grafana + the agent, pre-wired) and the full architecture + zero-egress verification live in the on-prem kit under deploy/onprem/. Air-gapped crash diagnoses are saved to a local incident log (DENPEX_INCIDENT_DIR), never transmitted. Fully-disconnected clusters mirror the image into their own registry.

SLURM & Ray

The agent is launched as a wrapper around your command, so it works under any scheduler with no per-scheduler setup. It auto-captures the scheduler's environment (SLURM job/node/proc id, Ray job/node id, K8s pod/namespace) so the dashboard shows exactly which job, node, and rank failed.

Proactive monitoring. Live, GPU telemetry, schedulers

Beyond wrapping a command, the agent bundle ships three drop-in tools that catch problems earlier and remove the manual paste entirely. Each is a single stdlib-only file.

The telemetry collector is read-only and low-overhead; most clusters already run DCGM-Exporter as part of the NVIDIA GPU Operator, so you can point Denpex at the metrics endpoint you already expose. Findings are diagnosed and routed to your alert channels, see the integrations guide for the live monitor and experiment-tracker connectors.

Privacy & data minimization

Masking runs client-side, before any byte leaves your cluster. Set the mode with DENPEX_PRIVACY:

• mask (default). Emails, IPs, MACs, SSNs, API keys, tokens, JWTs, credentials, and home/Windows paths are redacted before egress. Error text, stack traces, and Xid/CUDA/NCCL signatures are preserved (that's the diagnostic signal; it carries no PII).
• strict. Raw logs never leave. The agent extracts only anonymized failure signatures (e.g. Xid 79, CUDA out of memory, RuntimeError) and ships those. Recommended for regulated / PHI workloads.
• off. Send raw logs unchanged (opt-in).

Air-gapped / in-VPC (zero egress)

With DENPEX_LOCAL=1, the agent diagnoses failures entirely on the host using the same canonical deterministic pipeline as the hosted wrapper: 10,800+patterns, causal detectors, the Xid → node-action router, cross-rank cascade localization, and the 700+-entry encyclopedia. Nothing leaves the cluster: no logs, no signatures, no heartbeats, no API key. Only the LLM fallback and live research are unavailable offline.