High Availability Guide

← Back to reference

Scope: self-hosted / on-prem Kubernetes only. This guide describes the high-availability topology for customer-operated, self-hosted and air-gapped deployments. The Denpex SaaS control plane is single-region primary (WNAM) with globally-replicated reads and does not offer multi-region active-active — see docs/operations/SLA.md. Multi-region active-active, the active-active OTLP ingestion below, and cross-region replication apply only to self-hosted / on-prem installations.

Architecture

Three or five control-plane replicas behind a load balancer. Each replica has:

Its own SQLite (audit + feature store).
Shared S3-compatible object store.
Per-region KMS endpoint.

                    +-------------------+
                    | Load Balancer (TLS)|
                    +-------------------+
                       |        |        |
                       v        v        v
                  +-------+ +-------+ +-------+
                  | CP-1  | | CP-2  | | CP-3  |
                  +-------+ +-------+ +-------+
                       \        |        /
                        +-------+-------+
                                |
                                v
                  +--------------------------+
                  | Object Store (S3-compatible) |
                  +--------------------------+
                                |
                                v
                  +--------------------------+
                  | KMS (regional)           |
                  +--------------------------+

HA Tunables

Knob	Default	Notes
`hpa.minReplicas`	3	All replicas active at baseline
`hpa.maxReplicas`	9	Burst
`hpa.cpuTargetUtilization`	70 %
`leaderElection.enabled`	true	Only one replica processes audit writes
`leaderElection.leaseDurationSeconds`	30
`failureDomain`	zone-spread	Multi-AZ

Hot Spare

For the agent tier we maintain a pool of warm-spare nodes. The orchestrator probes them via Preflight before rerouting a job.

Data Plane HA

The agent is a DaemonSet; Kubernetes respawns any failed pod within terminationGracePeriodSeconds. State persists in /var/lib/denpex/agent/.

Routing

Active-passive for stateful workloads (only one replica processes audit writes).
Active-active for OTLP ingestion (stateless).
Round-robin via load balancer for REST.

Failover

If a CP replica fails:

The leader-elector (kube-rs Lease) elects a new leader within 30 s.
In-flight OTLP envelopes that point at the dead replica are retried by the agent.
The audit log is replayed from the shared WAL on the object store.

Cross-Region

For multi-region HA:

Each region runs its own CP replicas.
The agent connects to a regional endpoint.
Cross-region replication of audit logs uses async WAL shipping.

Recovery Objective

Objectives below are for self-hosted / on-prem deployment sizes and are not bound to SaaS pricing tiers. They assume a single-region control-plane cluster with the replica count shown; they do not imply multi-region recovery.

Self-hosted deployment size	RPO	RTO
Single-region, 3-replica cluster	15 min	1 h
Single-region, 5-replica cluster	5 min	30 min
Air-gap appliance	configurable	configurable

Validation

Quarterly game-days exercise: randomly kill 1 of 3 replicas and verify recovery.