Skip to content
Concepts

Core concepts

Denpex watches your distributed training jobs, catches failures as they happen, and tells you an evidence-ranked root cause and a concrete next action. Here is how the pieces fit together.

The agent

A single Python file that wraps your training command. Stdlib only, no dependencies, no kernel module, no root. It heartbeats every 2 minutes to report liveness, and on a non-zero exit it ships the last 500 log lines to the Denpex API. The agent never modifies the training process.

Three privacy modes control what leaves your cluster: mask(redact PII/PHI client-side before egress), strict(ship only anonymized failure signatures, raw logs never leave), and off (send raw logs unchanged).

For air-gapped clusters, DENPEX_LOCAL=1 runs the canonical deterministic pipeline on-host. No logs, signatures, or heartbeats leave the cluster; only hosted LLM and live-research enrichment are unavailable.

Diagnosis engine

The engine combines three detection tiers:

  • Deterministic patterns 10,800+ deterministic regex signatures that match known failure modes (NCCL timeouts, CUDA OOM, Xid errors, etc.)
  • Retrieval over the encyclopedia 700+ documented failure classes matched by IDF-weighted similarity with error-signature anchoring
  • AI fallback for novel errors that don't match any known pattern. Available on every plan within that plan's diagnosis allowance.

The engine outputs a root cause, a classification (hardware / ML / data / environment), the originating rank in distributed jobs, and the exact fix. For cascading failures, it correlates a single fabric event into one diagnosis instead of thousands of individual node alerts.

Alert channels

Diagnoses are routed to your configured alert channels: Slack, PagerDuty, SMS, email, or generic webhooks. Each channel can be configured with a minimum severity and quiet hours. On Team+ you can route by ownership so the right team gets notified.

Proactive monitoring

Beyond crash-only diagnosis, Denpex offers proactive monitoring that catches failures before the job crashes. The telemetry watcher polls NVML or DCGM-Exporter every 30 seconds and ships a diagnosis the instant it sees an Xid fault, uncorrectable ECC error, pending row-remap, or thermal throttle.

Metered pricing

Denpex bills on GPU-hours monitored, not per-seat. Every plan includes a base number of GPU-hours with overage pricing above that. Price protection guarantees existing subscribers keep their rate when list prices change. Annual plans save 2 months.