Remediation Api Reference

Endpoints

Run Remediation

POST /v1/<tenant_id>/remediation/run
Content-Type: application/json

{
  "host": "node-3",
  "rule_id": "DCGM_FI_DEV_XID_ERRORS=79",
  "severity": "critical",
  "warm_spare": "warm-spare-0",
  "step_timeout_ms": 30000,
  "tenant": "acme"  // optional override
}

Response:

{
  "incident_id": "uuid",
  "remediation_id": "uuid",
  "final_state": "COMPLETED",
  "rma_ticket": "RMA-...",
  "warm_spare": "warm-spare-0",
  "events": [
    { "action": "DetectXid", "success": true, "duration_ms": 1 },
    { "action": "CordonNode", "success": true, "duration_ms": 12 },
    { "action": "DrainSlurmNode", "success": true, "duration_ms": 17 },
    { "action": "TriggerCheckpoint", "success": true, "duration_ms": 8 },
    { "action": "EvictPod", "success": true, "duration_ms": 22 },
    { "action": "RerouteJob", "success": true, "duration_ms": 31 },
    { "action": "RunDcgmDiagnostics", "success": true, "duration_ms": 9800 },
    { "action": "ResumeFromCheckpoint", "success": true, "duration_ms": 19 },
    { "action": "GenerateRmaTicket", "success": true, "duration_ms": 4 }
  ]
}

Status

GET /v1/<tenant_id>/remediation/<id>

Rollback

POST /v1/<tenant_id>/remediation/<id>/rollback

Approve

POST /v1/<tenant_id>/remediation/<id>/approve
{
  "decision": "AutoRemediate",
  "actor": "alice@example.com"
}

Action Classes

Class	Default Tier
`Drain`	Approve
`Evict`	Approve
`Reroute`	Approve
`RunDiagnostic`	AutoSafe
`OpenRmaTicket`	AutoSafe
`RebootNode`	Approve
`ResetGpu`	Approve

Override via policy.yaml.

Audit

Every action emits an AuditRecord (see docs/audit-logging-guide.md). Hash chain verifier: denpex-control-plane audit-verify.

Air-Gap Behavior

In air-gapped mode, webhook side-effects are dropped unless explicitly allow-listed.