Core concepts

The agent

A single Python file that wraps your training command. Stdlib only, no dependencies, no kernel module, no root. It heartbeats every 2 minutes to report liveness, and on a non-zero exit it ships the last 500 log lines to the Denpex API. The agent never modifies the training process.

Three privacy modes control what leaves your cluster: mask(redact PII/PHI client-side before egress), strict(ship only anonymized failure signatures, raw logs never leave), and off (send raw logs unchanged).

For air-gapped clusters, DENPEX_LOCAL=1 runs the entire diagnosis engine on-host. No logs, signatures, or heartbeats leave the cluster at all.

Diagnosis engine

The engine combines three detection tiers:

• Deterministic patterns 1,400+ hand-tuned regex signatures (1,423 failure types) that match known failure types (NCCL timeouts, CUDA OOM, Xid errors, etc.)
• Retrieval over the encyclopedia 3500+ documented failure classes matched by IDF-weighted similarity with error-signature anchoring
• AI fallback for novel errors that don't match any known pattern. Available on paid plans.

The engine outputs a root cause, a classification (hardware / ML / data / environment), the originating rank in distributed jobs, and the exact fix. For cascading failures, it correlates a single fabric event into one diagnosis instead of thousands of individual node alerts.

Alert channels

Diagnoses are routed to your configured alert channels: Slack, PagerDuty, SMS, iMessage, email, or generic webhooks. Each channel can be configured with a minimum severity and quiet hours. On Team+ you can route by ownership so the right team gets notified.

Proactive monitoring

Beyond crash-only diagnosis, Denpex offers proactive monitoring that catches failures before the job crashes. The telemetry watcher polls NVML or DCGM-Exporter every 30 seconds and ships a diagnosis the instant it sees an Xid fault, uncorrectable ECC error, pending row-remap, or thermal throttle.