CI/CD gates: catch it before it burns GPU-hours

A misconfiguration that reaches a 512-GPU job costs thousands of dollars before the first NCCL timeout. These gates run the same single-file agent your training nodes use, on your CI runner, and fail the pipeline on any blocker, version known-bads, corrupt checkpoints, and (on GPU runners) the full NCCL topology lint. No API key. Nothing leaves the runner.

GitHub Action

Add one step to any workflow. The agent is a single file with no dependencies, so the gate is a curl and a python3. The subcommand selects the gate: preflight · preflight --deep · check-versions · validate-checkpoint.

# .github/workflows/training-gate.yml
name: training-gate
on: [pull_request]

jobs:
  denpex-preflight:
    runs-on: [self-hosted, gpu]     # any runner that can see the GPUs
    steps:
      - uses: actions/checkout@v4
      - name: Denpex preflight (deep NCCL topology lint)
        run: |
          curl -fsSL -o denpex.py https://denpex.com/agent/denpex.py
          python3 denpex.py preflight --deep --checkpoint-dir ./checkpoints
        # exits non-zero on any blocker, which fails the job

--deep adds the NCCL topology lint: GPU→NIC affinity, PCIe ACS, NCCL env validated against the runner's real fabric, and memlock ceilings. It runs the same engine as the free web tool at /preflight, offline and keyless, and a parity test in CI keeps the two from ever disagreeing. On a CPU-only hosted runner, swap the subcommand for check-versions (known-bad PyTorch × CUDA × driver combos) or validate-checkpoint ./checkpoints. Both run anywhere Python 3.8+ runs.

GitLab CI

Include the hosted template and pick the check with a variable:

# .gitlab-ci.yml
include:
  - remote: 'https://denpex.com/ci/denpex-preflight.gitlab-ci.yml'

denpex-preflight:
  stage: test
  tags: [gpu]                    # a runner that can see the GPUs
  variables:
    DENPEX_CHECK: preflight-deep
    DENPEX_CHECKPOINT_DIR: ""    # set for validate-checkpoint

What each gate catches

Gate	Blocks the pipeline when…	Runner
preflight	No GPUs visible, GPUs already held by zombie processes, incompatible torch × CUDA × driver, low disk, corrupt newest checkpoint.	GPU
preflight --deep	Everything in preflight, plus: PCIe ACS enabled (P2P/GDR killer), NCCL_IB_HCA referencing missing/Down HCAs, NCCL_SOCKET_IFNAME on docker0/lo or matching nothing, memlock ceilings that break ibv_reg_mr, GPU→NIC affinity warnings.	GPU
POST /api/preflight/topology (Scale+)	The same deep lint, server-side, when CI collects the bundle from the target cluster rather than the runner itself. Also free and account-less in the browser at /preflight.	any (posts a bundle)
check-versions	A known-bad PyTorch / CUDA / cuDNN / NCCL combination is about to ship.	any
validate-checkpoint	The newest checkpoint in the given directory is incomplete or unloadable (and tells you the last GOOD one).	any

API-side gate (Scale+)

If your CI collects a topology bundle from the target cluster (not the runner), post it to the same engine server-side and fail on summary.verdict === "fail":

curl -s https://api.denpex.com/api/preflight/topology \
  -H "Authorization: Bearer $DENPEX_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"artifacts\": $(jq -Rs . < topo-bundle.txt)}" \
| jq -e '.summary.verdict != "fail"'

The free in-browser version of the same engine lives at denpex.com/preflight.

Why this is worth a pipeline minute

The gate moves Denpex from the incident workflow into the development workflow: instead of diagnosing the NCCL timeout after it wasted a night of H100-hours, the merge that would have caused it never lands. One prevented misfire on a 64-GPU job pays for years of CI minutes.