← Back to reference

Storage Bottleneck Guide

What We Capture

Lustre

  • /proc/fs/lustre/*/jobstats: read_bytes, write_bytes, OST distribution, MDS RPC counts.
  • /proc/fs/lustre/llite/*/dump_page_cache.
  • lctl get_param osc.*.stats for OST throughput.

BeeGFS

  • beegfs-ctl --listnodes --mirror.
  • beegfs-ctl --storagepool --listnodes.
  • /proc/beegfs/*/conn_stats.

NFS

  • /proc/net/rpc/nfsd per-op stats.
  • nfsstat -s server-side.
  • /proc/self/mountstats per-mount.

S3

  • Latency per GetObject / PutObject.
  • 5xx throttle counts.

Rules

RuleTrigger
OST_RETRY_STORMretry count > 50 / min
MDT_P99_LATENCY_HIGHMDT p99 > 4× baseline
CHECKPOINT_WRITE_SLOWcheckpoint write > 30s p99
CHECKPOINT_INVENTORY_DRIFTshard newer than manifest
S3_THROTTLED5xx > 1 % over 5 min

Common Symptoms and Causes

Symptoms: NCCL collectives stall mid-job

  • Often a saturated OST or MDT. The OST p99 rule fires.
  • Cross-reference with the rank that first started stalling.

Symptoms: checkpoint save stalls at step 5k

  • Often Lustre OST saturation. Inspect job stats.
  • Run lctl dl to see active connections.

Symptoms: S3 throttle during checkpoint

  • Storage rule fires; check S3_THROTTLED.
  • Recommend checkpointing to a dedicated bucket or moving to a different prefix.

Local-Side

We monitor PSI / cgroup on the agent host:

  • /proc/pressure/cpu and /proc/pressure/memory.
  • Cgroup memory high exhaustion.
  • Cgroup memory.events.low / high / oom counters.

When the local disk is exhausted, denpex raises StorageEvent::LocalCgroupExhausted and surfaces the GPU host as the source.

See Also

  • docs/data-collection-matrix.md#checkpoint-inventory.
  • docs/checkpoint-integrity-guide.md.