Privacy Model
What We Collect
| Class | Examples | Justification |
|---|---|---|
| Process telemetry | GPU kernel duration, NCCL AllReduce wall-clock | Required for gray-failure detection. |
| Per-layer weight L2 / gradient L2 / checksum | Per-step summary statistics | Required for SDC/collapse detection. Tensor contents are never exported. |
| Stdout / stderr | Wrapped process logs, truncated to last 10MB | Required for first-failed-rank identification. |
| NCCL topology XML | Once per job | Required for ring/tree/CollNet identification. |
What We DO NOT Collect
- Training data tensors.
- Optimizer state contents.
- Model weights.
- User prompts, completion text, or chat history.
- PII unless the user explicitly opts in via a
--with-piiflag.
Truncation + Hashing
- Logs are truncated at 10MB before forwarding to the control plane.
- User identifiers (Linux UIDs) are hashed with HMAC-SHA-256 using a per-tenant salt.
- Hostnames are kept in plaintext (required for remediation) but are masked in any export sent to a third-party SIEM.
Data Residency
Each tenant picks a residency at ingest time:
- US (default SaaS)
- EU (Frankfurt + Dublin regions)
- AP (Tokyo + Singapore)
- CUSTOMER_VPC (only ever leaves the customer's VPC, never reaches our SaaS)
Retention
See docs/data-pipeline-architecture.md#retention.
Right to Erasure
Tenants can request erasure via /v1/<tenant_id>/erase. The control plane emits an audit record, then physically deletes all per-tenant shards within 30 days. Backups are erased within 90 days. Erasure logs are themselves audited.
Per-Region Encryption Keys
Each region's KMS key is independent. A region compromise does not leak data from other regions.