Skip to content
Self-hosted

Self-hosted agent (in-VPC)

For data centers that cannot let telemetry leave their network, run the Denpex platform on your own hardware. Two deployment models: air-gapped (zero egress, no API key required) and cloud mode (diagnoses report to api.denpex.com from behind your firewall).

Helm chart (Kubernetes)

The Denpex Helm chart ships a pinned container image with the agent baked in. Deploy as a DaemonSet (one pod per GPU node) with Prometheus metrics on port 9836.

# Add the chart repo
helm repo add denpex https://denpex.com/charts
helm repo update

# Air-gapped mode: nothing leaves the cluster, no API key
helm install denpex denpex/denpex-agent --set airGapped=true

# Cloud mode: agents report to api.denpex.com
helm install denpex denpex/denpex-agent --set airGapped=false --set apiKey=dpx_...

# Metrics endpoint
# denpex_up / denpex_gpus_total / denpex_anomalies / denpex_gpu_anomaly

Air-gapped (zero egress)

With DENPEX_LOCAL=1, the agent diagnoses failures entirely on the host using a bundled copy of the engine: the regex pattern tier, the Xid to node-action router, cross-rank cascade localization, and the full 360-class encyclopedia matched by IDF-weighted similarity. No logs, signatures, or heartbeats leave the cluster. The LLM fallback is cloud-only and simply unavailable offline; the deterministic engine covers the large majority of real failures locally.

# Download the agent and offline engine (once) curl -O https://denpex.com/agent/denpex.py curl -O https://denpex.com/agent/denpex_local.py curl -O https://denpex.com/agent/denpex_patterns.json # Run fully offline DENPEX_LOCAL=1 python denpex.py run --job llama3 -- torchrun --nproc_per_node=8 train.py # Or diagnose a saved log file DENPEX_LOCAL=1 python denpex.py diagnose-local ./crash.log

Monitoring stack

In air-gapped mode, fleet health is served as Prometheus metrics. Point your existing Prometheus or VictoriaMetrics at each node's :9836/metrics endpoint. A starter stack with Prometheus, VictoriaMetrics, and Grafana pre-wired is included in the on-prem kit.

Air-gapped crash diagnoses are saved to a local incident log (DENPEX_INCIDENT_DIR) and never transmitted. Fully disconnected clusters mirror the container image into their own registry.

Prerequisites

  • Python 3.8+ on each node (stdlib only, no additional packages)
  • Kubernetes 1.24+ for the Helm chart, or any Linux host for the standalone agent
  • For air-gapped mode: download denpex_local.py and denpex_patterns.json alongside the agent
  • Network access to https://denpex.com/agent/ for the initial download (or mirror the files internally)
  • NVIDIA GPU with appropriate drivers (the agent reads GPU info via NVML)