Skip to content
Agent

Install the agent

The Denpex agent is a single Python file. It wraps your training command, heartbeats every 2 minutes, and ships the last 500 log lines on a non-zero exit. Stdlib only, no dependencies, no kernel module, no root.

Install & verify

# 1. download the agent (single file) + the signed checksum manifest
curl -O https://denpex.com/agent/denpex.py
curl -O https://denpex.com/agent/SHA256SUMS

# 2. verify integrity before you run anything on your cluster
sha256sum --ignore-missing -c SHA256SUMS
# denpex.py: OK

# 3. read it — it's one file, stdlib only, ~1,000 lines
less denpex.py

Python 3.8+. No third-party packages, no kernel module, no root, no daemon. The agent never modifies the training process and never installs anything. It's a single readable file — verify the checksum, read the source, then run it.

Preview exactly what leaves your cluster (before you trust it)

You don't have to take our word for it. DENPEX_DRY_RUN=1 prints the exact, already-masked payload that would be sent and transmits nothing. DENPEX_PRIVACY=strict ships only anonymized failure signatures (raw logs never leave), and DENPEX_LOCAL=1 runs the whole engine on-host with zero egress.

DENPEX_DRY_RUN=1 DENPEX_PRIVACY=strict python denpex.py run -- python train.py

Wrap a training command

export DENPEX_API_KEY=dpx_...

# PyTorch DDP / FSDP / DeepSpeed
python denpex.py run --job llama3-finetune -- torchrun --nproc_per_node=8 train.py

# Megatron-LM
python denpex.py run --job megatron-70b -- bash megatron/run_pretrain.sh

# Single-GPU
python denpex.py run --job debug -- python train.py --batch-size 32

Environment variables

DENPEX_API_KEY*
API key from your dashboard. Required.
DENPEX_PRIVACY
mask (default) — redact PII/PHI client-side before egress · strict — ship only anonymized failure signatures, raw logs never leave · off — send raw logs.
DENPEX_DRY_RUN
Set to 1 to print the EXACT (masked) payload that would be sent and send nothing. Audit egress before trusting the agent with sensitive logs.
DENPEX_HEARTBEAT_SEC
Cloud liveness heartbeat interval in seconds. Default 120 (clamped 5–3600). Fault detection stays frequent regardless — this only throttles the liveness ping, which cuts request volume ~4x at fleet scale.
DENPEX_TAIL_LINES
Lines to ship on crash. Default 500, max 5000.
DENPEX_LOCAL
Set to 1 for AIR-GAPPED mode — diagnosis runs entirely on-host and monitoring is served as local Prometheus metrics; no logs, signatures, or heartbeats leave. Needs denpex_local.py + denpex_patterns.json next to denpex.py. No API key required.
DENPEX_METRICS_PORT
Expose a Prometheus /metrics endpoint on this port (denpex_up, denpex_gpus_total, denpex_anomalies, denpex_gpu_anomaly). Auto-on at 9836 in air-gapped mode; set explicitly to enable in cloud mode too.
DENPEX_INCIDENT_DIR
Air-gapped mode: directory for local incident JSON logs (one per on-host diagnosis). Default ./denpex-incidents.
DENPEX_API_URL
Override the API base URL.

Kubernetes

Run the agent as a sidecar container. The agent shares the pod's filesystem and lifecycle with your training container; no extra scheduling is needed.

- name: training image: your-training-image:latest command: ["python", "/denpex/denpex.py", "run", "--job", "$(JOB_NAME)", "--"] args: ["torchrun", "--nproc_per_node=8", "train.py"] - name: denpex-agent image: denpex/agent:latest env: - name: DENPEX_API_KEY valueFrom: secretKeyRef: name: denpex-secret key: api-key volumeMounts: - name: shared-logs mountPath: /logs

Self-hosted / air-gapped (Helm)

For data centers that can't let telemetry leave their network, run the whole platform on your own hardware with one command. The agent diagnoses failures on-host and serves fleet health as Prometheus metrics — your own Prometheus + Grafana watch the cluster, with zero egress and no per-request cost. Ships as a DaemonSet (one pod per GPU node) from the Denpex on-prem kit.

# 1. add the chart repo (once)
helm repo add denpex https://denpex.com/charts
helm repo update

# 2a. air-gapped — nothing leaves the cluster, no API key
helm install denpex denpex/denpex-agent --set airGapped=true

# 2b. or cloud mode — agents report to api.denpex.com
helm install denpex denpex/denpex-agent --set airGapped=false --set apiKey=dpx_...

# the agent then serves :9836/metrics — point your Prometheus at it:
denpex_up · denpex_gpus_total · denpex_anomalies · denpex_gpu_anomaly{gpu,code,severity}

The chart ships a pinned, scripts-baked-in container image — no repo checkout, no manual ConfigMap. A runnable starter stack (Prometheus + VictoriaMetrics + Grafana + the agent, pre-wired) and the full architecture + zero-egress verification live in the on-prem kit under deploy/onprem/. Air-gapped crash diagnoses are saved to a local incident log (DENPEX_INCIDENT_DIR), never transmitted. Fully-disconnected clusters mirror the image into their own registry.

SLURM & Ray

The agent is launched as a wrapper around your command, so it works under any scheduler with no per-scheduler setup. It auto-captures the scheduler's environment (SLURM job/node/proc id, Ray job/node id, K8s pod/namespace) so the dashboard shows exactly which job, node, and rank failed.

# SLURM (sbatch) — wrap srun's payload srun python denpex.py run --job $SLURM_JOB_NAME -- torchrun --nproc_per_node=8 train.py # Ray — wrap the entrypoint of each worker ray job submit -- python denpex.py run --job $RAY_JOB_ID -- python train.py

Proactive monitoring — live, GPU telemetry, schedulers

Beyond wrapping a command, the agent bundle ships three drop-in tools that catch problems earlier and remove the manual paste entirely. Each is a single stdlib-only file.

# 1. Live early-warning — catch NaN / grad-spike / divergence as it happens
curl -O https://denpex.com/agent/denpex_live.py
# from denpex_live import LiveMonitor / DenpexLiveCallback (auto-checkpoints on divergence)

# 2. Native GPU telemetry — read DCGM-Exporter or NVML directly (Xid, ECC, thermal, row-remap)
curl -O https://denpex.com/agent/denpex_telemetry.py
python denpex_telemetry.py --dcgm http://localhost:9400/metrics --watch
python denpex_telemetry.py --nvml # read the driver directly on-node

# 3. Zero-touch scheduler hooks — auto-diagnose FAILED SLURM / Kubernetes jobs
curl -O https://denpex.com/agent/denpex_scheduler.py
curl -O https://denpex.com/agent/denpex-epilog.sh # SLURM: EpilogSlurmctld=/opt/denpex/denpex-epilog.sh

The telemetry collector is read-only and low-overhead; most clusters already run DCGM-Exporter as part of the NVIDIA GPU Operator, so you can point Denpex at the metrics endpoint you already expose. Findings are diagnosed and routed to your alert channels — see the integrations guide for the live monitor and experiment-tracker connectors.

Privacy & data minimization

Masking runs client-side, before any byte leaves your cluster. Set the mode with DENPEX_PRIVACY:

  • mask (default) — emails, IPs, MACs, SSNs, API keys, tokens, JWTs, credentials, and home/Windows paths are redacted before egress. Error text, stack traces, and Xid/CUDA/NCCL signatures are preserved (that's the diagnostic signal; it carries no PII).
  • strict — raw logs never leave. The agent extracts only anonymized failure signatures (e.g. Xid 79, CUDA out of memory, RuntimeError) and ships those. Recommended for regulated / PHI workloads.
  • off — send raw logs unchanged (opt-in).
# preview EXACTLY what would leave (sends nothing)
DENPEX_DRY_RUN=1 DENPEX_PRIVACY=strict python denpex.py run -- python train.py

Air-gapped / in-VPC (zero egress)

With DENPEX_LOCAL=1, the agent diagnoses failures entirely on the hostusing a bundled copy of the engine — the regex pattern tier, the Xid → node-action router, cross-rank cascade localization, and the full 3500+-class encyclopedia matched by IDF-weighted similarity + error-signature anchoring. Nothing leaves the cluster: no logs, no signatures, no heartbeats, no API key. The LLM fallback is cloud-only and is simply unavailable offline; the deterministic engine (which covers the large majority of real failures) runs locally.

# download the agent + its offline engine + pattern DB (once) curl -O https://denpex.com/agent/denpex.py curl -O https://denpex.com/agent/denpex_local.py curl -O https://denpex.com/agent/denpex_patterns.json # run fully offline — diagnosis prints on-host, nothing is transmitted DENPEX_LOCAL=1 python denpex.py run --job llama3 -- torchrun --nproc_per_node=8 train.py # or diagnose a saved log file offline DENPEX_LOCAL=1 python denpex.py diagnose-local ./crash.log

What gets shipped

  • Heartbeat (job name, hostname, status, timestamp) every 30 seconds.
  • On non-zero exit: the last 500 log lines, after client-side masking, with the failure type inferred from exit code.
  • For hangs (no heartbeat for 3+ minutes): the agent exits with code 137, and the orchestration stack records the failure.
  • Nothing else. We do not read source code, model weights, datasets, environment variables, or any other files.