CI/CD gates: catch it before it burns GPU-hours

A misconfiguration that reaches a 512-GPU job costs thousands of dollars before the first NCCL timeout. These gates run the same single-file agent your training nodes use, on your CI runner, and fail the pipeline on any blocker — version known-bads, corrupt checkpoints, and (on GPU runners) the full NCCL topology lint. No API key. Nothing leaves the runner.

GitHub Action

Add one step to any workflow. check selects the gate: preflight · preflight-deep · check-versions · validate-checkpoint.

# .github/workflows/training-gate.yml
name: training-gate
on: [pull_request]

jobs:
  denpex-preflight:
    # use your self-hosted GPU runner for the deep topology lint
    runs-on: [self-hosted, gpu]
    steps:
      - uses: actions/checkout@v4
      - name: Denpex preflight (deep NCCL topology lint)
        uses: denpex/preflight-action@v1
        with:
          check: preflight-deep
          checkpoint-dir: ./checkpoints   # optional

On a CPU-only hosted runner, use check: check-versions (known-bad PyTorch × CUDA × driver combos) or validate-checkpoint — both run anywhere Python 3.8+ runs.

GitLab CI

Include the hosted template and pick the check with a variable:

# .gitlab-ci.yml
include:
  - remote: 'https://denpex.com/ci/denpex-preflight.gitlab-ci.yml'

denpex-preflight:
  stage: test
  tags: [gpu]                    # your GPU runner, for preflight-deep
  variables:
    DENPEX_CHECK: preflight-deep
    DENPEX_CHECKPOINT_DIR: ""    # set for validate-checkpoint

What each gate catches

Gate	Blocks the pipeline when…	Runner
preflight	No GPUs visible, GPUs already held by zombie processes, incompatible torch × CUDA × driver, low disk, corrupt newest checkpoint.	GPU
preflight-deep	Everything in preflight, plus: PCIe ACS enabled (P2P/GDR killer), NCCL_IB_HCA referencing missing/Down HCAs, NCCL_SOCKET_IFNAME on docker0/lo or matching nothing, memlock ceilings that break ibv_reg_mr, GPU→NIC affinity warnings.	GPU
check-versions	A known-bad PyTorch / CUDA / cuDNN / NCCL combination is about to ship.	any
validate-checkpoint	The newest checkpoint in the given directory is incomplete or unloadable (and tells you the last GOOD one).	any

API-side gate (Scale+)

If your CI collects a topology bundle from the target cluster (not the runner), post it to the same engine server-side and fail on summary.verdict === "fail":

curl -s https://api.denpex.com/api/preflight/topology \
  -H "Authorization: Bearer $DENPEX_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"artifacts\": $(jq -Rs . < topo-bundle.txt)}" \
| jq -e '.summary.verdict != "fail"'

The free in-browser version of the same engine lives at denpex.com/preflight.

Why this is worth a pipeline minute

The gate moves Denpex from the incident workflow into the development workflow: instead of diagnosing the NCCL timeout after it wasted a night of H100-hours, the merge that would have caused it never lands. One prevented misfire on a 64-GPU job pays for years of CI minutes.