Data Pipeline Architecture

End-to-end pipeline from agent ingest → storage → query → action.

Stages

Agent (OTLP) ──> Receiver ──> Validate ──> Route
                                          │
   ┌──────────────────────────────────────┼──────────────────────┐
   v                                       v                       v
Time-series store                  Log store                Trace store
(redb+mmap or Influx)              (loki/REST or S3)        (tempo/REST or parquet)
   │                                                                       │
   └────────────────────┐                          ┌───────────────────────┘
                        v                          v
                   Feature store (SQLite) ─────► Object store (S3/GCS/Azure)
                              │
                              v
                       Graph DB (petgraph)
                              │
                              v
                   EvidenceIndex ───► retrieval (lancedb)
                              │
                              v
                   Diagnosis + Remediation

Per-Tenant Data Partitioning

Each tenant has:

• A separate EntityHistory table prefix (tenant_<id>_*).
• A separate evidence-index segment.
• A separate OTLP receiver route mounted at /v1/<tenant_id>/....
• Row-level filtering enforced at the query boundary.

Tenant ids are baked into every record's metadata; the agent attaches them at ingest time based on the API token claims.

Retention

Downsampling runs as a background task; results are stored in separate time-series tables with a lifecycle tag.

Backpressure

If the time-series store can't keep up, the receiver drops new samples and increments denpex_pipeline_dropped_total. The control plane emits a Prediction::TelemetryPipelineDegraded when this counter rises persistently.