← Back to reference

Network Failure Guide

What Denpex Monitors

InfiniBand

  • Per-port counters (symbol_error, port_xmit_wait, link_downed, link_error_recovery, VL15_dropped, etc.)
  • Per-VL counters for QoS analysis.
  • ibdiagnet-shaped topology walk.
  • ibstat / ibqueryerrors integration.
  • UFM REST integration when UFM_ENDPOINT env var is set.

RoCEv2

  • ethtool -S per-prio PFC + counter stats.
  • mlxlink module telemetry.
  • devlink eswitch QoS.
  • CNP detection (cnp_rcv, cnp_send, rp_ecn_marked).
  • gNMI/SONiC via --roce-gnmi flag.
  • sFlow/NetFlow export parsing.

Ethernet

  • TCP retransmit via tcp:tcp_retransmit_skb eBPF probe.
  • NIC link flaps via sysfs.
  • PSI pressure.

GPUDirect / RDMA

  • /sys/module/nvidia_peermem/version
  • modinfo nvidia_peermem
  • dmesg | grep peer
  • NCCL env NCCL_IB_HCA, NCCL_P2P_LEVEL, NCCL_NET_GDR_LEVEL

Rules

RuleTrigger
IB_LINK_ERROR_RECOVERYlink_error_recovery > 0
ROCE_CNP_BURSTcnp_rcv rate above threshold
ROCE_PAUSE_BURSTrx_pause_frames above threshold
NCCL_NET_SOCKET_FALLBACKAny rank uses NET/Socket/0
NVLINK_REPLAYreplay count > 0 / hour
SWITCH_SXID_CRITICALSXid 22013
GIGABIT_LOOPBACK_FAILPCIe loopback failed

Common Symptoms and Causes

Symptoms: NCCL collective stalls at scale

  1. Open Denpex incident.
  2. Check IB_LINK_DOWNED_PORT. If > 0, look at the IB counter file /sys/class/infiniband/mlx5_X/ports/1/counters/*.
  3. Cross with NVLink counters.

Symptoms: RoCE CNP storm

  1. Check ROCE_CNP_BURST and rx_pause_frames.
  2. Inspect the per-prio ethtool counters.
  3. Adjust PFC watchdog or move the workload.

Compatibility Tested

  • mlx5 driver 5.x and 6.x.
  • Mellanox ConnectX-5/6/7.
  • Broadcom PEX89048.

See Also

  • docs/user-guides/nccl-debugging-guide.md.
  • docs/supported-network-matrix.md.