Supported Network Matrix
| Network | Status | Notes |
|---|---|---|
| RoCEv2 | Full | ethtool, mlxlink, devlink, CNP |
| Ethernet (TCP) | Best effort | PSI + TCP retransmit eBPF |
| SONiC / gNMI | Full | gNMI/JSON-IETF subscriptions |
| UFM | Full | REST |
| Fabric Manager | Full | /fm/v2/nvlinks, /fm/v2/sxid/events |
Per-network Notes
InfiniBand
Per-port counters: symbol_error, port_xmit_wait, link_downed, link_error_recovery, excessive_buffer_overrun, port_rcv_remote_physical_errors, local_link_integrity_errors, VL15_dropped, plus per-VL counters.
ibdiagnet / ibstat / ibqueryerrors integrated.
RoCEv2
ethtool -S <iface> for per-prio PFC and counter stats, mlxlink -p <dev> for module telemetry, devlink dev <iface> eswitch show for switch QoS.
CNP detection: cnp_rcv, cnp_send, rp_ecn_marked, rp_ecn_handled.
gNMI/SONiC switch collector behind --roce-gnmi (uses tonic + protobuf).
sFlow/NetFlow export parsing for loss + retransmit.
GPUDirect / RDMA
/sys/module/nvidia_peermem/version, modinfo nvidia_peermem, dmesg | grep peer, NCCL env NCCL_IB_HCA, NCCL_P2P_LEVEL, NCCL_NET_GDR_LEVEL.
Compatibility Tested
- mlx5 driver 5.x and 6.x.
- Mellanox ConnectX-5/6/7.
- Broadcom PEX89048 switches.
- Cisco N9K switches (gNMI).
Unsupported
- Pre-mlx5 hardware (CX-3) — basic sysfs only.
- Non-Mellanox RDMA (AWS EFA, libfabric-based) — best effort.