Skip to content
Quickstart

Prove Denpex on your first incident

Three paths. Path A is fastest. Paste a log, get a diagnosis, no install. Path B is for clusters, wrap your training command with the agent and submit failures for investigation. Path C watches available GPU telemetry. Recommendations require workload verification.

Path A. Zero-Config CLI Test Drive

Diagnose your first crash from the terminal without signing up.

  1. Install the current website distribution in your workload environment:
    python -m pip install "https://denpex.com/agent/releases/denpex-1.1.28-py3-none-any.whl#sha256=3dc3460e21bc19924767fb874f4da51ec65794677e335420c159683e3ebafbd0"
  2. Pass your failing log to the CLI:
    denpex diagnose training.log
  3. Review the evidence, recommendation and missing facts. Confirm the action applies to your versions and workload before changing it.

The first no-key diagnosis is frictionless. Later headless diagnoses require an API key; the browser check is available on the homepage. T0 (anonymous or negatively classified): 3/day. T1 (identified): 10/day. T2 (current verified organization proof): 50/day. T3 (active trial): the persisted trial cap, normally 50/day. T4 (paid): unlimited.

Path B. Connect one workload

The website distribution includes the agent and its evidence, replay and verification helpers. It wraps your training command and heartbeats every 120 seconds (DENPEX_HEARTBEAT_SEC). Its local GPU telemetry scrape runs more often, every 30 seconds, and never leaves your node.

# 1. Install the current complete agent distribution
python -m pip install "https://denpex.com/agent/releases/denpex-1.1.28-py3-none-any.whl#sha256=3dc3460e21bc19924767fb874f4da51ec65794677e335420c159683e3ebafbd0"

# 2. Set the API key from your dashboard
export DENPEX_API_KEY=dpx_...

# 3. Wrap your training command, that's it
python -m denpex run --job llama3-finetune -- torchrun --nproc_per_node=8 train.py

On a non-zero exit, the agent ships the last 500 log lines to the Denpex API and you get the diagnosis in your dashboard. Missing heartbeats flag unverified workload state; they do not independently prove a crashed job. If delivery fails, queued evidence is replayed when the agent next contacts the service.

Path C. Catch failures before the crash

Path B diagnoses a run after it dies. Path C watches the GPUs themselves. The telemetry watcher polls NVML. Or an existing DCGM-Exporter. Every 30 seconds and ships a diagnosis the instant it sees an Xid fault, uncorrectable ECC error, pending row-remap, or thermal throttle. That is usually minutes before the NCCL watchdog fires and the job actually crashes. By the time you open your laptop, the evidence and an investigation may already be available in your configured channel. Hardware access and diagnostic evidence determine what can be established.

# Use the telemetry watcher included in the installed distribution
export DENPEX_API_KEY=dpx_...

# For NVML, install the Python binding on each node
python -m pip install nvidia-ml-py
python -m denpex_telemetry --nvml --watch

# ...or scrape a DCGM-Exporter you already run
python -m denpex_telemetry --dcgm http://localhost:9400/metrics --watch

# See it right now: inject a simulated Xid 79 and watch the full loop
python -m denpex_telemetry --simulate

--simulate fires a synthetic GPU fault so you can watch detection → diagnosis → alert end-to-end in a single cycle, even on a laptop with no real fault. The fastest way to see the proactive loop work. Run one watcher per node, or point it at a rack-level DCGM-Exporter. Live fleet monitoring is included on Team and higher plans.

What happens next

  • Diagnosis lands in your dashboard with root cause + fix.
  • If you've configured an alert channel (Slack, PagerDuty, SMS, webhook), the diagnosis is pushed there with severity mapping.
  • When a fix works, press the thumbs-up under "Did this fix work?". It goes into your team's knowledge base against that failure class, and the next time anyone on your team hits it, that fix leads the diagnosis. The thumbs-up on the diagnosis itself is a separate control for whether the diagnosis was right.

Next steps