Prometheus / Grafana metrics export

Denpex exposes a standard /metrics endpoint in Prometheus exposition format, so failures, MTTR, Xid events, per-node reliability and cost rollups show up in the dashboards your SRE team already lives in — Grafana, Datadog, or any OpenTelemetry collector.

The endpoint

The exporter lives on the API worker at /metrics. It is authenticated — use your Denpex API key as a Bearer token (or X-Denpex-Key header, the same key the agent uses). It only ever emits your own tenant's data.

# Bearer token
curl -H "Authorization: Bearer dpx_..." \
  https://api.denpex.com/metrics

# or the X-Denpex-Key header (agent style)
curl -H "X-Denpex-Key: dpx_..." \
  https://api.denpex.com/metrics

Exported metrics

Metric	Type	Description
denpex_failures_total	counter	Total diagnosed failures in the 30-day scrape window
denpex_failures_by_type	counter	Failures grouped by failure class (label: type)
denpex_mttr_ms	gauge	Mean time-to-root-cause across recent diagnoses
denpex_gpu_hours_lost_total	counter	Measured GPU-hours burned by diagnosed failures
denpex_cost_usd_total	counter	Estimated $ of compute lost to diagnosed failures
denpex_xid_events_total	counter	NVIDIA Xid events observed in retained logs (label: code)
denpex_jobs_total	gauge	Observed jobs (any status)
denpex_jobs_running	gauge	Jobs currently reporting heartbeats
denpex_node_runs_total	counter	Runs observed per node (label: node)
denpex_node_failures_total	counter	Crashed/failed runs per node (label: node)
denpex_node_reliability	gauge	1 − failures/runs per node; 1.0 = perfectly reliable

Scrape config (Prometheus)

Add a scrape job to your prometheus.yml. The API key goes in an authorization header — never inline it in a shared config.

scrape_configs:
  - job_name: 'denpex'
    scrape_interval: 60s
    scheme: https
    authorization:
      credentials: dpx_your_api_key_here
    static_configs:
      - targets: ['api.denpex.com']
    metrics_path: /metrics

Grafana dashboard

Useful PromQL panels to start with:

# Failures per day (rate)
sum(rate(denpex_failures_total[1d])) * 86400

# Mean time to root cause
denpex_mttr_ms / 1000

# GPU-hours burned this month
denpex_gpu_hours_lost_total

# Top failing nodes
topk(5, denpex_node_failures_total)

# Node reliability (worst first)
bottomk(5, denpex_node_reliability)

# Xid events by code
sum by (code) (denpex_xid_events_total)

OpenTelemetry collector

To forward into Datadog, New Relic, or any OTLP backend, use the prometheusreceiver in the OTel collector:

receivers:
  prometheus/denpex:
    config:
      scrape_configs:
        - job_name: denpex
          scrape_interval: 60s
          metrics_path: /metrics
          authorization:
            credentials: ${env:DENPEX_API_KEY}
          static_configs:
            - targets: ['api.denpex.com']

service:
  pipelines:
    metrics:
      receivers: [prometheus/denpex]
      exporters: [otlp/dem-backend]

Notes & limits

The scrape window is 30 days; the exporter returns up to 500 recent diagnoses and jobs per scrape.
Node attribution is best-effort: it reads node/hostname from job metadata, then from log previews. Unknown nodes bucket as unknown.
The cost rollups (denpex_gpu_hours_lost_total, denpex_cost_usd_total) only include incidents where the runtime was measured (the agent ships this; a pasted-log diagnosis reports a burn rate instead).
For the fleet reliability UI (worst nodes, RMA candidates), see /api/fleet/reliability in the API reference.