Telemetry Schema Reference

Per-Emission Schema

Kind	Source	Cardinality	Sample
`dcgm.dbe_total`	DCGM	1/GPU	`dcgm.dbe.total, gpu.uuid`
`dcgm.row_remap_pending`	DCGM	1/GPU	`dcgm.row_remap.pending, gpu.uuid`
`dcgm.pcie_replay_total`	DCGM	1/GPU	`dcgm.pcie_replay.total, gpu.uuid`
`dcgm.nvlink_replay_total`	DCGM	1/NVLink	`dcgm.nvlink_replay.total, gpu.uuid, link.id`
`nvml.power_usage`	NVML	1/GPU	`nvml.power_usage, gpu.uuid`
`nvml.thermal_throttle_count`	NVML	1/GPU	`nvml.throttle.thermal, gpu.uuid, hour`
`nvml.ecc.volatile.sbe`	NVML	1/GPU	`nvml.ecc.sbe.volatile, gpu.uuid`
`ib.symbol_error`	IB sysfs	1/port	`ib.symbol_error, dev, port`
`ib.link_downed`	IB sysfs	1/port	`ib.link_downed, dev, port`
`ib.link_error_recovery`	IB sysfs	1/port	`ib.link_error_recovery, dev, port`
`roce.pfc.rx_pause`	ethtool	1/iface	`roce.pfc.rx_pause, iface, prio`
`roce.cnp`	mlx5 sysfs	1/iface	`roce.cnp_rcv, iface`
`pcie.gen.current`	NVML	1/GPU	`pcie.gen.current, gpu.uuid`
`pcie.width.current`	NVML	1/GPU	`pcie.width.current, gpu.uuid`
`nccl.allreduce.seconds`	NCCL	1/rank/op	`nccl.allreduce.seconds, job_id, rank, op`
`layer.weight_l2`	SDK	1/step/layer	`layer.weight_l2, layer, step`
`layer.grad_l2`	SDK	1/step/layer	`layer.grad_l2, layer, step`
`layer.nan_count`	SDK	1/step/layer	`layer.nan_count, layer, step`
`layer.inf_count`	SDK	1/step/layer	`layer.inf_count, layer, step`
`psi.cpu.full_avg10`	/proc	1/host	`psi.cpu.full_avg10, host`
`psi.memory.full_avg10`	/proc	1/host	`psi.memory.full_avg10, host`
`psi.io.full_avg10`	/proc	1/host	`psi.io.full_avg10, host`
`cgroup.memory.events.low`	cgroup	1/cgroup	`cgroup.memory.events.low, cgroup`
`ebpf.event.count`	ringbuf	1/program	`ebpf.event.count, program`

Naming Conventions

Prefix: source — nvml., dcgm., ib., roce., nccl., pcie., layer., psi., cgroup., ebpf., agent..
Suffix: unit (total, seconds, bytes, count, celsius).

Units

We adopt OpenTelemetry base units:

s for seconds.
By for bytes.
1 for dimensionless counts.

Retention

Default retention per metric is 7 days raw; downsampled at 30 d to 1 minute, 90 d to 1 hour, 365 d to 1 day. Retention is configured per tenant in policy.yaml.

Telemetry Schema Reference

Per-Emission Schema

Naming Conventions

Units

Retention

See Also