← Back to reference

Slurm Deployment Guide

Prerequisites

  • SLURM 22.05 or 23.02 (sinfo -V)
  • slurmrestd enabled and reachable
  • The denpex service account on every compute node

Step 1: Install the agent RPM/DEB

sudo rpm -i denpex-enterprise-agent-*.rpm
# or
sudo apt install ./denpex-enterprise-agent_*.deb

The package:

  • Creates user denpex-agent.
  • Installs binaries to /usr/lib/denpex/.
  • Writes systemd unit denpex-agent.service.
  • Installs /etc/denpex/agent.yaml.

Step 2: Configure /etc/denpex/agent.yaml

denvex:
  slurm:
    endpoint: https://slurmrestd.example.com:6820
    token: ${slurm_token}
    verify: true
    caBundle: /etc/denpex/slurm-ca.pem
  otlp:
    endpoint: https://denpex-cp.example.com:4317
    token: ${agent_token}
  probes:
    pcie: true
    ib: true
    gpu: true

Step 3: Install the SPANK plugin

sudo rpm -i denpex-spank-*.rpm
# writes /opt/slurm/lib/slurm/spank_denpex.so
echo "required /opt/slurm/lib/slurm/spank_denpex.so" \
    | sudo tee -a /etc/slurm/plugstack.conf
sudo systemctl restart slurmctld

Step 4: Start the agent

sudo systemctl enable --now denpex-agent.service
sudo journalctl -fu denpex-agent

Step 5: Submit a test job

sbatch --wrap="sleep 30" --gres=gpu:8 \
    --export=ALL,DENPEX_TENANT=acme \
    test.sh

The agent captures the job start, rank-to-host mapping, comm binding, and GPU state for each rank.

Configuration Reference

See slurm/modules/spank_denpex.c for the full SPANK context captured.

Compatibility Tested

  • SLURM 22.05.7, 23.02.6.
  • RHEL 9.4, Rocky 9.4, Ubuntu 22.04 LTS.

Migration from a non-Denpex site

docs/operations/slurm-migration.md covers:

  • preserving existing job IDs.
  • preserving existing topology.
  • coexistence with existing monitoring.