← Back to reference

Architecture Whitepaper

The 7-layer reliability platform for AI training infrastructure.

Layered Stack

+-------------------------------------------------------------------+
| Layer 7 | Enterprise Plane      | SAML, OIDC, RBAC, KMS, Audit     |
| Layer 6 | Control Plane         | Correlation, diagnosis, RAG       |
| Layer 5 | Remediation           | State machine, policy, audit       |
| Layer 4 | Training SDK          | Per-layer metrics, SDC canary      |
| Layer 3 | Fabric Collector      | IB, RoCE, NCCL, GPUDirect          |
| Layer 2 | GPU Collector         | NVML, DCGM, CUPTI                  |
| Layer 1 | Host Agent            | eBPF, PSI, journald, NUMA          |
| Layer 0 | denpex_lite           | stdlib Python wrapper, signatures  |
+-------------------------------------------------------------------+

Design Goals

  1. Deterministic first. Every diagnosis is backed by a rule whose firing is reproducible from the same evidence. AI is summarization only.
  2. Air-gapped first. Every release artifact installs in a fully offline environment with no PyPI/Crates/Helm pull at runtime.
  3. Tenant-isolated by construction. RBAC + audit + per-tenant data partitioning are evaluated before any operator query reaches the storage layer.
  4. Provable claims. Every published number is reproducible from a benchmark in benchmarks/ or a test in this repository.

Layer Responsibilities

  • Layer 0 (denpex_lite): Wrap a Python training process with a stdlib-only PTY, capture stdout/stderr, parse 397 deterministic failure signatures, build a causal timeline, emit AI summary.
  • Layer 1 (denpex-enterprise-agent): Long-running host agent that loads eBPF programs on sched:sched_switch, oom:mark_victim, tcp:tcp_retransmit_skb, page-fault tracepoints, and uprobes on libcuda.so / libnccl.so. Reads PSI, cgroup, procfs, sysfs, journald, NUMA. Emits OTLP envelopes.
  • Layer 2 (GPU Collector): real FFI bindings to NVML and DCGM via libnvidia-ml.so.1 and libdcgm.so. Falls back to shim parser only when DENPEX_NVML_SHIM=1 (tests).
  • Layer 3 (Fabric Collector): IB sysfs per-port counters, RoCE per-prio PFC stats, NCCL topology XML, GPUDirect state, UFM/gNMI/Fabric Manager REST clients.
  • Layer 4 (Training SDK): PyTorch forward/backward hooks record per-layer weight_l2, grad_l2, NaN/Inf, XXH3 checksum, canary SDC microbatch. DeepSpeed/Megatron/FSDP/JAX/TF integrations.
  • Layer 5 (Remediation): 10-step state machine: detect → cordon → drain → preStop checkpoint → evict → reroute → DCGM Level-3 → resume → RMA → done. Tiered policy (Observe/Recommend/ Approve/AutoSafe/AutoRemediate), append-only audit with hash chain, preflight probes on warm-spare.
  • Layer 6 (Control Plane): Causality graph, 7+ deterministic rules, confidence scoring, retrieval-augmented diagnostic summary via local sentence-transformer embedding store.
  • Layer 7 (Enterprise Plane): SAML/OIDC/SCIM/MFA, RBAC + ABAC, KMS via AWS/GCP/Azure, Vault + K8s secrets + HSM, multi-tenant data partitions, SOC2/ISO27001/HIPAA/FedRAMP controls.

Deployment Topologies

  • Single cluster SaaS: denpex-control-plane hosted, denpex_lite + denpex-enterprise-agent installed on each node, RMA webhooks integrated with Slack/PagerDuty/Jira.
  • Multi-cluster SaaS: denpex-control-plane replicates state per region; per-tenant data partitioning enforced at ingest.
  • Air-gapped appliance: All artifacts distributed as a signed tarball (denpex-airgap-bundle.tar.gz); install.sh verifies kernel ≥ 5.15, loads signed BPF programs, bootstraps local object store + S3-compatible endpoint, configures HSM-rooted KMS.

References