Skip to content

Deployment architecture

Put GPU failure diagnosis on the correct side of your boundary

Denpex supports connected, minimized-egress and fully local diagnostic paths. The right architecture depends on who operates the GPUs, what evidence may leave, and whether the workflow stops at diagnosis or includes a separately qualified action receiver.

The common data flow

  1. 1

    Workload

    Training, inference, scheduler or saved log

  2. 2

    Collect

    Agent, API, editor, hook or local file

  3. 3

    Minimize

    Mask, strict signatures or local-only path

  4. 4

    Diagnose

    Order evidence, classify owner, propose control

  5. 5

    Verify

    Readback, outcome, metrics and approved alerting

Deployment paths

Connected agent

Fit
Teams that can send masked incident evidence to the hosted diagnosis API.
Data boundary
Client-side masking runs before logs leave. Strict signature-only mode is also available.
Result
Hosted deterministic diagnosis, evidence-permitted deeper analysis, tenant incident history and alerts.

In-VPC agent

Fit
Private GPU platforms that require collection and selected processing inside their cloud boundary.
Data boundary
Deployment and egress rules are agreed with the eligible plan and reviewed against the target VPC.
Result
Agent, monitoring and approved diagnosis components run inside the customer-controlled network path.

Air-gapped local mode

Fit
HPC, defense, laboratory and regulated environments with no outbound service dependency.
Data boundary
No logs, signatures, heartbeats or API key leave in DENPEX_LOCAL mode.
Result
Bundled deterministic diagnosis, local incident JSON and Prometheus metrics. Hosted fallback and live research are unavailable.

Controlled execution pilot

Fit
Qualified environments evaluating governed remote actions after diagnosis and shadow-mode validation.
Data boundary
Requires a reviewed receiver, short-lived scoped authorization, signed directives and explicit policy.
Result
Dry run, authorization, execution reporting, readback verification and rollback. This is not implied by the standard agent.

Architecture by segment

Each segment changes the evidence source and operating owner, even when the diagnosis engine is shared.

Recommended Denpex architecture considerations by GPU infrastructure segment.
SegmentEvidence intakePrimary outputDeployment note
GPU cloudJob wrapper plus node telemetryTenant-scoped incidents, GPU UUID history, fleet metricsStart connected or in-VPC; qualify control-plane actions separately
Inference platformServing worker logs, request shape metadata, GPU telemetryEngine, kernel, capacity and host ownershipConnected or in-VPC, with canary verification before traffic return
ML platformAgent, API, editor and scheduler intakeShared diagnosis and verification record across research teamsConnected, strict or in-VPC based on data classification
Frontier trainingAll-rank logs, scheduler events, checkpoint evidenceInitiating rank, cascade, resume evidence and measured lossScoped pilot on the real launcher, topology and checkpoint path
HPC and SlurmJob, step, node, GRES, cgroup and local framework evidenceLocal runbooks, Prometheus and administrator-owned node stateAir-gapped or connected, with no required root for basic wrapping
OEM and RMAXid, ECC, GPU identity, recurrence and controlled test evidenceRMA candidate, field diagnosis or not-RMA-worthy packetConnected or local evidence preparation; vendor remains the warranty authority

Boundaries to put in the design review

  • The root layout and agent do not require a second privileged control plane for saved-log diagnosis.
  • The standard public agent monitors, diagnoses and supplies human-run steps. It does not imply hands-off remote command authority.
  • Local deterministic mode cannot provide hosted fallback or live external research.
  • A deployment diagram does not establish security or compliance qualification for a customer environment. Procurement review remains required.

Continue the technical review

Use the public docs for the implementation path, then request the deeper data-flow and security materials for procurement.

Self-hosted docsProcurement checklist