API

REST API reference

Base URL: https://api.denpex.com. Authenticated requests require a Bearer token or API key. The explicitly documented anonymous test-drive route is the only diagnosis exception. See the agent docs for how to create a key, or generate one in your dashboard.

Authentication

# All authenticated requests

Authorization: Bearer dpx_...

Tokens are JWTs issued by the auth worker. They expire 30 days after issue. Rotation is supported by creating a new key and deleting the old one.

# Or, with a dpx_ API key (the MCP server and CI use this)

X-Denpex-Key: dpx_...

The API resolves the user and plan from the key server-side, so no JWT minting is needed. Either header works on every endpoint below.

Streaming (SSE)

POST /api/diagnose with { stream: true } returns text/event-stream instead of JSON. This is a different response shape, not a different encoding of the same one: the engine emits its reasoning as it works, then one terminal frame carrying the full result. Read to the terminal frame, the thoughts are narration, not the answer.

# zero or more, as the engine reasons

event: thought

data: {"text":"Checking for cross-rank cascade…"}

# exactly one of these ends the stream

event: result

data: { ...the same envelope the non-streaming call returns }

event: error

data: {"error":"…"}

Streamed diagnoses are persisted, alerted on, and given a real diagnosisId exactly like non-streamed ones, so feedback and history work the same way. Idempotency-Key does not apply to streams, replaying a stale stream would be worse than re-running it.

Endpoints

/api/diagnose/anon
None
Zero-signup test drive used by the homepage and no-key CLI/agent command. It delegates to the same canonical diagnosis worker as /api/diagnose, but does not persist history or enable account-scoped features. Limited to 3 successful diagnoses per UTC day per privacy-preserving IP token; failed engine calls do not consume a run. A successful response includes an opaque refinementToken. To answer clarifyingChoices without spending another diagnosis, send the same original logs and full operatorAnswers history with X-Denpex-Refinement: <refinementToken>. The token is bound to that log and browser/network fingerprint; an action selection is a request to collect evidence, not the result of the check.
Request
```
{ logs: string, operatorAnswers?: Array<{ questionId?, optionId?, kind?, question?, answer }> }
```
Response
```
{ success: true, remaining, refinementToken, engineVersion, fix: { summary, action, steps?, fallbackAction?, emergencyAction?, verificationCommand?, prevention?, evidence?, causationChain?, competingHypotheses?, clarifyingChoices?, evidenceRequests?, awaitingEvidence? }, meta: { failureType, confidence, source, tier, engineVersion } }
```

`/api/diagnose`

Bearer token

Diagnose a training log. Returns root cause, classification, originating rank (if distributed), and the exact fix. Send `Idempotency-Key` to make retries safe: a repeat within 10 minutes replays the first response (header `Idempotency-Replayed: true`) without re-running the engine or consuming quota. Set `stream: true` for Server-Sent Events instead, see the streaming note below.

Request

{ logs: string, jobName?: string, stream?: boolean, telemetry?: { dmesg?, nvidia_smi?, ibstat? }, codeContext?: Record<string,string>, topologyMap?: Record<string,string[]>, runtimeHours?: number, gpuRate?: number, operatorAnswers?: Array<{ questionId?, optionId?, kind?, question?, answer }> }

Response

{ success: true, diagnosisId, initiator?, diagnosisTimeMs, jobName, advanced?, priorIncident?, cost, costHeadline, costAdvisor?, advisory?, contradictions?, rankedHypotheses?, falsificationReport?, forensicReport?, communityRefs?, researchRefs?, semanticRefs?, originNote?, telemetryEvidence?, resumeSafetyNote?, secondaryHardwareNote?, falsePaths?, causalityDisclaimer?, confidenceBand?, confidenceDisplay?, fix: { summary, action, steps?, fallbackAction, emergencyAction, verificationCommand, prevention, resumeFrom?, evidence, causationChain, competingHypotheses, alternatives?, clarifyingQuestions, clarifyingChoices?, evidenceRequests?, awaitingEvidence?, inviteFollowup, confidenceBand?, confidenceDisplay? }, meta: { failureType, confidence, source, tier } }

/api/diagnose/history
Bearer token
Recent diagnoses for the authenticated user, newest first. `limit` defaults to 20 and is capped at 50. Paginate with `?cursor=<nextCursor>`: keep passing the `nextCursor` from the previous page until it comes back null. Free plans see the last 7 days; Team+ is unlimited.
Request
```
query: ?limit=20&cursor=<ms-timestamp>
```
Response
```
{ history: Diagnosis[], nextCursor: string | null, hasMore: boolean, limit: number }
```
/api/diagnose/quota
Bearer token
Plan, remaining diagnoses today, GPU cap, and the shipped corpus size. Free is capped at 3/day; paid plans return unlimited.
Response
```
{ quota: { allowed: boolean, remaining: number }, plan, gpuLimit, failureTypes, signatures, deterministicPatterns }
```
/api/diagnose/feedback
Bearer token
Report whether a fix worked. This is the only input to the engine's learning loop, a reported outcome reweights the next diagnosis of the same failure class. `note` matters most on a did_not_work.
Request
```
{ diagnosisId: string, outcome: 'worked' | 'did_not_work', note?: string, newFix?: string }
```
Response
```
{ success: true }
```
/api/alerts/resolve
Account session only
Explicitly close one incident after a human operator checks the recovery evidence. This manual override accepts an authenticated account session only, not an API key. Denpex records the operator and reason, closes the exact tenant-owned incident first, then sends resolution notifications using the incident's server-stored identity. Automated closure is available only through a persisted passing verification report.
Request
```
{ incidentId: string, diagnosisId: string, humanConfirmed: true, reason: string }
```
Response
```
{ success: true, incidentsClosed: number }
```
/api/incidents/ack
Bearer token
Acknowledge an incident: 'I'm on it, stop paging me.' Alerts for it are suppressed until it is resolved. Identify it by whichever handle you hold.
Request
```
{ incidentId? | diagnosisId? | incidentKey?, by?: string }
```
Response
```
{ success: true, acknowledged: number, acknowledgedBy: string }
```
/api/incidents/snooze
Bearer token
Silence an incident for a bounded window, then resume paging automatically. Use it for 'the RMA is in flight, not until Monday.' Defaults to 60 minutes, capped at a week.
Request
```
{ incidentId? | diagnosisId? | incidentKey?, minutes?: number }
```
Response
```
{ success: true, snoozed: number, snoozedUntil: number, minutes: number }
```
/api/ingest/heartbeat
Bearer token
Agent heartbeat. Reports job status every 2 minutes by default (configurable with DENPEX_HEARTBEAT_SEC). A terminal beat with status: "completed" may carry the run's launch fingerprint, which is stored as the baseline a later failure of the same job is diffed against, a job that has never finished successfully has nothing to compare a failure to.
Request
```
{ jobName, hostname, status, ts, fingerprint? }
```
Response
```
{ ok: true, runId, baselineRecorded }
```
/api/ingest/crash
Bearer token
Agent crash report. Ships the last 500 log lines on non-zero exit. Mass-crash safe: reports sharing a cluster + failure signature within 5 minutes coalesce into ONE diagnosis. Followers return the shared fix with coalesced: true and consume no quota. Optional fingerprint and evidenceBundle add the launch-time run fingerprint and the termination evidence the agent captured; with them the response also carries evidence (which physical GPU the failing rank held, which ranks NCCL RAS found unresponsive, hardware counters, checkpoint resumability, and what could not be captured this time) and environmentDiff against the last successful run of the same job. A follower's evidence is recorded too when it carries a causal signature, in a fabric event the rank holding the Xid is rarely the one elected leader. See Rate limits for details.
Request
```
{ runId?, jobName, logs, exitCode, meta?, fingerprint?, evidenceBundle? }
```
Response
```
{ success: true, diagnosisId, coalesced?, crashCount?, fix: { summary, action, fallbackAction?, emergencyAction?, resumeFrom? }, meta: { failureType, confidence }, evidence?, environmentDiff? }
```
/api/verification/report
Bearer token; Idempotency-Key header required
Record whether a fix actually worked. Send the readback checks from `denpex verify --report`; the server applies its own required closure plan and returns one of verified_recovered, awaiting_observation, partially_recovered, failed_verification, recurrence_detected or inconclusive. Use a new Idempotency-Key for each observation and the same key only to retry that exact observation. Each decisive pass or fail includes observedAt so stale spooled evidence cannot be applied to a newer recovery window. The elapsed window is still measured from the server's FIRST sighting, never from a client timestamp. The validated fixAppliedAt value only groups reports from the same recovery attempt: a new value starts a new server window, and its numeric clock value never shortens that window. A required failure or a recurrence is accepted immediately; waiting cannot disprove those. An exact tenant-linked incident closes only after the passing report is durably stored and its node, GPU, and linked job still match. Unreferenced reports are retained for 90 days; evidence referenced by a retained incident or remediation run, and evidence for an active diagnosis, is retained with that record.
Request
```
{ checks: [{ id, status, detail?, observedAt }], fixAppliedAt, diagnosisId?, node?, gpuUuid?, jobName? }
```
Response
```
{ id, stored, outcome, summary, mayCloseRun, incidentsClosed, score, scoreLabel, anchorAt, observedForS, recheckInS, passed, failed, pending, unavailable }
```
/api/verification/outcomes
Bearer token
How often the fixes Denpex recommends actually work, tallied per outcome with a mean effectiveness score over ?days (default 30). Below ten reports the response carries a note saying the sample is too small to read as a rate, rather than rounding four data points into a percentage.
Response
```
{ days, total, outcomes: [{ outcome, count, meanScore }], note: string | null }
```
/api/remediation/run/{id}/report
Bearer token; Idempotency-Key header required for phase verified
Advance one authorized remediation run through started, executed, verified, rollback_started, or rollback_done. A verified report requires an Idempotency-Key header, a stable fixAppliedAt value reused across polls for that recovery attempt, and structured checks from the requested server plan. Use a new key for each observation and the same key only to retry that exact observation. The legacy cleared boolean is not accepted as recovery evidence. A required failure can move a reversible run into rolling_back and return its rollback commands; only a durably stored complete passing report can verify the run and close its exact incident.
Request
```
{ phase: 'started' | 'executed' | 'verified' | 'rollback_started' | 'rollback_done', fixAppliedAt (required for verified), exit_code?, output?, detail?, checks?: [{ id, status, detail?, observedAt }] }
```
Response
```
{ run_id, state, terminal, incidentsClosed, rollbackCommands?, verification?, verificationReportId?, adversePersistence?, history }
```
/api/fleet/evidence
Bearer token
Read back the incident evidence the agent captured. Pass ?diagnosis=<id> for every rank that contributed to one incident (leader first), or ?gpu=<uuid> for everything one physical card has been involved in. The card view counts distinct incidents rather than reports, and sets hasLowConfidenceAttribution when any report tied that serial number to a rank by device index rather than authoritatively, a history assembled partly from inference should not read like a certainty when it becomes an RMA. Paid plans.
Response
```
{ evidence: IncidentEvidence[], card: { gpuUuid, gpuSerial, incidentCount, firstSeen, lastSeen, signatures, hasLowConfidenceAttribution } | null }
```
/api/fleet/reliability-report
Bearer token
The reliability review for a window of ?days (default 30): mean time to remediate, mean time to verified recovery, remediation success, recurrence and false-remediation rates, GPU-hours lost, top failure classes and repeat offenders. Every metric declares whether it was measured or estimated, and every estimate carries the formula and assumptions behind it so the arithmetic can be re-run; override them with ?gpuHourCostUsd and ?assumedManualDetectionMinutes. A metric with no supporting events returns null with an unavailableReason rather than a zero, notably mean-time-to-detect, which we do not claim because we record when we were told a job died, not when it died. Simulated failures from /api/monitor/simulate are excluded. Add ?format=csv for the export. Distinct from /api/fleet/reliability, which is the per-node Xid/ECC and RMA-candidate view. Paid plans.
Response
```
{ metrics: [{ id, label, kind: 'measured'|'estimated', value: number|null, unit, sampleSize, formula?, assumptions?, unavailableReason? }], incidentCount, topFailureClasses, repeatOffenders, windowDays, estimatesTraceable }
```
/api/keys
Bearer token
List API keys for the authenticated user.
Response
```
{ keys: ApiKey[] }
```
/api/keys
Bearer token
Create a new API key. The full key is returned only once.
Request
```
{ label?: string }
```
/api/jobs
Bearer token
List jobs the agent has reported heartbeats for.
Response
```
{ jobs: Job[] }
```
/api/alerts/channels
Bearer token
List alert channels for the authenticated user.
Response
```
{ channels: AlertChannel[] }
```

`/api/alerts/channels`

Bearer token

Add a new alert channel.

Request

{ channelType: 'slack' | 'webhook' | 'sms' | 'pagerduty' | 'email', config: object, enabled?: boolean }

/api/alerts/channels
Bearer token
Remove an alert channel.
Request
```
{ id: number }
```
/api/billing/checkout
Bearer token
Start a Stripe Checkout Session for a paid plan.
Request
```
{ plan: 'team' | 'scale' }
```
Response
```
{ url: string }
```
/api/billing/subscription
Bearer token
Current subscription, including plan, status, and period end.
Response
```
{ plan, subscription: { id, plan, status, currentPeriodEnd, cancelAtPeriodEnd }, trial: { ... } | null }
```
/api/billing/portal
Bearer token
Open the Stripe customer portal for self-service plan changes and invoice downloads.
Response
```
{ url: string }
```
/api/billing/webhook
Stripe signature
Stripe webhook. Handles checkout.session.completed and customer.subscription.* events.
/api/redeem
Bearer token
Redeem a trial code. Each account may redeem one code.
Request
```
{ code: string }
```
/api/preflight/topology
Bearer token / API key
Deep NCCL topology lint (Scale+): GPU→NIC affinity, PCIe ACS, NCCL_IB_HCA / NCCL_SOCKET_IFNAME validated against the pasted fabric, memlock. Same engine as the free /preflight web tool, this endpoint exists for CI gates.
Request
```
{ artifacts: string }  // one blob: nvidia-smi topo -m, env, ibstat, ip -br addr, lspci ACSCtl, ulimit -l
```
Response
```
{ success: true, findings: [{ id, severity, title, detail, fix, evidence? }], summary: { blockers, warnings, infos, verdict } }
```
/api/diagnose/env-diff
Bearer token / API key
Cross-run environment diff (Team+). Extracts driver/CUDA/NCCL/framework/packages/env from artifacts of a good and a failing run, ranks every change by correlation with the observed failure class.
Request
```
{ goodRun: string, badRun: string, failureLogs?: string }
```
Response
```
{ success: true, headline, failureClass, primarySuspect, suspects: [...], changes: [{ component, from, to, correlation, rationale }] }
```
/api/cost/advisor
Bearer token / API key
Cost optimization recommendations from logs: evidence-gated right-sizing, spot↔on-demand reliability trades, utilization advice. Also attached automatically to every diagnosis as `costAdvisor`.
Request
```
{ logs: string, gpus?: number, failureType?: string, ratePerGpuHour?: number }
```
Response
```
{ success: true, recommendations: [{ kind, headline, detail, estMonthlyUsd, confidence, basis }], detected: {...} }
```
/api/fleet/history
Bearer token / API key
Per-GPU telemetry history, 1-minute rollups (min/max/last per counter), 90-day retention. This is the 'what did node-1234 GPU 3 look like six hours before it died' query. `from`/`to` are epoch ms and default to the last 6 hours; omit `gpu` for every device on the node plus node-level scalars. `truncated: true` means the window hit the per-query object cap and the answer is short, treat it as short. If `available` is false, this deployment has no telemetry bucket bound yet and `howToEnable` says what to do. Scale+ feature.
Request
```
query: ?node=node-1234&gpu=3&from=<ms>&to=<ms>
```
Response
```
{ available: boolean, node, gpu, from, to, points: [{ t, node, gpu, m: { <counter>: { min, max, last } } }], pointCount, objectsRead, truncated }
```
/api/fleet/node-health
Bearer token / API key
Predictive pre-crash node health (Scale+, GA): per-node 0-100 score from live ECC/thermal trajectories, Xid/NVLink counters and incident recurrence, with trend, action, and a failure ETA only when a real trend supports one.
Response
```
{ summary: { nodes, healthy, watch, degrading, critical, unknown }, nodes: [{ host, score, band, action, trend, etaHeartbeats, signals: [...] }] }
```
/api/fleet/multicloud
Bearer token / API key
Multi-cloud single pane (Scale+): every monitored host grouped by provider (AWS/GCP/Azure/Lambda/CoreWeave/RunPod/on-prem/…) with GPU totals, degraded counts, and open Denpex-identified issues per group.
Response
```
{ totals: { providers, hosts, gpus, openIssues }, providers: [{ provider, label, hosts, gpus, degraded, openIssues, topIssues: [...] }] }
```
/api/remediation/rollback-plan
Bearer token / API key
Checkpoint rollback + resume planning (Scale+). Builds verify → cordon → resume-from-verified-checkpoint from a stored diagnosis. Never deletes checkpoints; data-corruption classes stay human-gated. execute:true is accepted only for explicitly qualified remote-execution pilots; otherwise the response contains the operator-run plan and an execution-unavailable explanation.
Request
```
{ diagnosis_id: string, node?: string, execute?: boolean, format?: 'md' }
```
Response
```
{ success: true, plan: { mode, eligible, steps: [...], resumeCommand, guardrails }, dispatched, runbook? }
```
/api/reports/handoff
Bearer token / API key
On-call shift handoff report (Team+): failures, resolutions, remediation actions, open incidents with the NEXT fix per escalation level, and a watchlist. ?hours=24 (1-168); &format=md returns Slack-ready Markdown.
Response
```
{ windowHours, kpis: {...}, watchlist: [...], pending: [...], resolved: [...], remediations: [...] }
```
/api/remediation/webhook
Bearer token / API key
Controlled-pilot endpoint for a qualified agent receiver. General-availability deployments return REMOTE_EXECUTION_UNAVAILABLE and do not register a command callback. Pilot receivers use per-tenant HMAC signing and a secret returned once at registration or rotation.
Request
```
{ url: 'https://…', rotate?: boolean }
```
Response
```
{ success: true, url, secret?: 'dpxw_…', secretPreview, agentSetup }
```
/api/reliability-index
None
The GPU Reliability Index dataset (public, no auth): cited field studies plus the documented failure surface per GPU family, interconnect and provider stack. Human-readable at denpex.com/gpu-reliability.
Response
```
{ corpus: { byGpuFamily, byInterconnect, byProviderStack... }, fieldData: { studies: [...] } }
```

REST API reference

Authentication

Streaming (SSE)

Endpoints

`/api/diagnose/anon`

`/api/diagnose`

`/api/diagnose/history`

`/api/diagnose/quota`

`/api/diagnose/feedback`

`/api/alerts/resolve`

`/api/incidents/ack`

`/api/incidents/snooze`

`/api/ingest/heartbeat`

`/api/ingest/crash`

`/api/verification/report`

`/api/verification/outcomes`

`/api/remediation/run/{id}/report`

`/api/fleet/evidence`

`/api/fleet/reliability-report`

`/api/keys`

`/api/keys`

`/api/jobs`

`/api/alerts/channels`

`/api/alerts/channels`

`/api/alerts/channels`

`/api/billing/checkout`

`/api/billing/subscription`

`/api/billing/portal`

`/api/billing/webhook`

`/api/redeem`

`/api/preflight/topology`

`/api/diagnose/env-diff`

`/api/cost/advisor`

`/api/fleet/history`

`/api/fleet/node-health`

`/api/fleet/multicloud`

`/api/remediation/rollback-plan`

`/api/reports/handoff`

`/api/remediation/webhook`

`/api/reliability-index`