Prometheus / Grafana metrics export

Denpex exposes a standard /metrics endpoint in Prometheus exposition format, so failures, MTTR, Xid events, per-node reliability and cost rollups show up in the dashboards your SRE team already lives in. Grafana, Datadog, or any OpenTelemetry collector.

The endpoint

The exporter lives on the API worker at /metrics. It is authenticated. Use your Denpex API key as a Bearer token (or X-Denpex-Key header, the same key the agent uses). It only ever emits your own tenant's data.

# Bearer token
curl -H "Authorization: Bearer dpx_..." \
  https://api.denpex.com/metrics

# or the X-Denpex-Key header (agent style)
curl -H "X-Denpex-Key: dpx_..." \
  https://api.denpex.com/metrics

Exported metrics

Metric	Type	Description
denpex_failures_total	counter	Total diagnosed failures in the 30-day scrape window
denpex_failures_by_type	counter	Failures grouped by failure class (label: type)
denpex_mttr_ms	gauge	Mean time-to-root-cause across recent diagnoses
denpex_gpu_hours_lost_total	counter	Measured GPU-hours burned by diagnosed failures
denpex_cost_usd_total	counter	Estimated $ of compute lost to diagnosed failures
denpex_xid_events_total	counter	NVIDIA Xid events observed in retained logs (label: code)
denpex_jobs_total	gauge	Observed jobs (any status)
denpex_jobs_running	gauge	Jobs currently reporting heartbeats
denpex_node_runs_total	counter	Runs observed per node (label: node)
denpex_node_failures_total	counter	Crashed/failed runs per node (label: node)
denpex_node_reliability	gauge	1 − failures/runs per node; 1.0 = perfectly reliable

Scrape config (Prometheus)

Add a scrape job to your prometheus.yml. The API key goes in an authorization header. Never inline it in a shared config.

scrape_configs:
  - job_name: 'denpex'
    scrape_interval: 60s
    scheme: https
    authorization:
      credentials: dpx_your_api_key_here
    static_configs:
      - targets: ['api.denpex.com']
    metrics_path: /metrics

Grafana dashboard

Useful PromQL panels to start with:

# Failures per day (rate)
sum(rate(denpex_failures_total[1d])) * 86400

# Mean time to root cause
denpex_mttr_ms / 1000

# GPU-hours burned this month
denpex_gpu_hours_lost_total

# Top failing nodes
topk(5, denpex_node_failures_total)

# Node reliability (worst first)
bottomk(5, denpex_node_reliability)

# Xid events by code
sum by (code) (denpex_xid_events_total)

OpenTelemetry collector

To forward into Datadog, New Relic, or any OTLP backend, use the prometheusreceiver in the OTel collector:

receivers:
  prometheus/denpex:
    config:
      scrape_configs:
        - job_name: denpex
          scrape_interval: 60s
          metrics_path: /metrics
          authorization:
            credentials: ${env:DENPEX_API_KEY}
          static_configs:
            - targets: ['api.denpex.com']

service:
  pipelines:
    metrics:
      receivers: [prometheus/denpex]
      exporters: [otlp/dem-backend]

Notes & limits

The scrape window is 30 days; the exporter returns up to 500 recent diagnoses and jobs per scrape.
Node attribution is best-effort: it reads node/hostname from job metadata, then from log previews. Unknown nodes bucket as unknown.
The cost rollups (denpex_gpu_hours_lost_total, denpex_cost_usd_total) only include incidents where the runtime was measured (the agent ships this; a pasted-log diagnosis reports a burn rate instead).
For the fleet reliability UI (worst nodes, RMA candidates), see /api/fleet/reliability in the API reference.