Data Collection Matrix
| Data | Source | What is Collected | Where it Lands | Retention | Optional? |
|---|---|---|---|---|---|
| Fabric telemetry | IB sysfs, RoCE ethtool/devlink | Symbol errors, link down, RNR, PFC pause frames, CNP | OTLP -> Time-series store | Same as GPU | No |
| NCCL topology | NCCL XML on each rank | Parsed XML once per job | Topology graph (per job) | 30 d | No |
| Per-layer metrics | denpex_sdk.instrument | Weight L2/Linf, grad L2/Linf, NaN/Inf count, checksum | OTLP -> Per-layer anomaly engine | 7 d raw, 90 d 1m | Yes (can be disabled per job) |
| Checkpoint inventory | notify-rs on a path | Object key, sha256, size | Storage inventory + audit | 90 d | Yes |
| eBPF events | sched/oom/tcp/page-fault | PID, comm, address, latency | Ringbuf -> OTLP -> log store | 24h raw | No |
| Host PSI / cgroup / procfs | /proc/pressure, /proc/cgroup | some_avg10, full_avg10, failcnt | Time-series | 7 d raw | No |
| Stdout/stderr of wrapped proc | process stderr | truncated 10MB per rank | Log store | 7 d raw | No |
| Optimization state moments | Adam m/v means | per-rank per-step | Per-layer store | 30 d | Yes |
| Logs (kubelet, journald) | libsystemd-sys | streamed via cursor | Loki / local | 30 d | No |
| Crackdowns on K8s events | kube watch | pod, pvc, node event JSON | Trace store | 30 d | No |
User-Tunable
In denpex_lite, opt-out flags:
--no-layer-metrics— disables per-layer collection.--no-fabric— disables IB/RoCE polling (only valid for non-NCCL jobs).--no-bpf— disables eBPF (degrades to journald-only).
In denpex-enterprise-agent, similar flags in /etc/denpex/agent.yaml.