← Back to reference

Remediation Architecture

The remediator runs a 10-step state machine triggered by every DiagnosisEvent whose action class resolves to a non-Observe automation tier. It is built on top of two backends:

  • KubernetesBackend — abstraction over the kube crate (real) or SimulatedKubernetes (tests/offline).
  • SlurmBackend — abstraction over slurmrestd (real) or SimulatedSlurm (tests/offline).

State Machine

Detected
   |
   v
NodeCordoned     (kube.cordon_node)
   |
   v
CheckpointTriggered  (kube.trigger_pre_stop_checkpoint)
   |
   v
PodEvicted       (kube.evict_pods via policy/v1)
   |
   v
JobRerouted      (slurm.reroute_job_to_warm_spare after preflight)
   |
   v
DiagnosticsCompleted (kube.run_dcgm_diagnostics Level-3 on warm-spare)
   |
   v
Completed        (kube.resume_from_checkpoint)
   |
   v
RmaIssued        (kube.generate_rma_ticket)
   |
   v
Completed (terminal) OR Failed (terminal)

Each step emits a RemediationEvent and the orchestrator records an AuditRecord with hash-chained prev/current hashes.

Preflight

Before rerouting onto a warm-spare, the orchestrator runs Preflight { PcieLoopback, IbPingPong, GpuMemcpy, NvSwitchBw }. Any failed probe aborts the reroute and a DiagnosisEvent is raised against the warm-spare node.

Postflight

After eviction, Postflight { PcieLoopback, GpuMemcpy, DiskSequentialWrite } is run on the original node. Results are recorded but do not block the state machine — the node is considered quarantined.

Policy

PolicyEngine resolves ActionClass -> AutomationTier based on a YAML policy:

policy:
  - action: cordon
    tier: auto_remediate
    conditions:
      - kind: severity
        at_least: critical
  - action: drain
    tier: approve
    conditions:
      - kind: tenant
        id: acme

Tiers:

  • Observe — log only.
  • Recommend — emit recommendation, no execution.
  • Approve — emit + wait for human approval.
  • AutoSafe — auto-execute, no impact (diagnostics, RMA).
  • AutoRemediate — auto-execute.

Audit Log

Append-only, hash-chained. audit-verify walks the chain and reports any broken link.

Air-Gap

When air_gapped=true, the orchestrator drops any webhook/HTTP side-effect that isn't in the explicit allow-list. Side effects include Slack, PagerDuty, Jira, S3 puts, KMS signs.