Pending and rejected jobs
Resolve job reason codes against the complete CPU, memory, GPU, feature, reservation and QOS request.
For HPC centers and research computing
HPC and Slurm GPU cluster reliability
HPC support teams need to distinguish an impossible job request from a drained-node problem, a cgroup step failure from application OOM, and a scheduler symptom from the distributed job that failed first.
Denpex connects Slurm job, step, node and GRES evidence with framework, NCCL, CUDA and GPU telemetry. That gives administrators a shared runbook for user support while keeping local policy and node-health decisions under cluster control.
Who this is for
- HPC systems administrators
- Research computing engineers
- Slurm administrators
- GPU cluster engineers
- University AI platform teams
- National laboratory infrastructure teams
The failure surface
These incidents look adjacent in the final alert, but they require different owners and different proof before recovery.
GRES and rank mapping
Explain job-local GPU visibility and detect duplicate launcher or device-ordinal assumptions.
Cgroup and host memory
Separate job-step host OOM from framework-reported CUDA device OOM.
Drained node decisions
Preserve scheduler, health-check, Xid and ECC evidence before clearing or escalating a GPU node.
The operating loop
- 1
Read scheduler truth
Capture scontrol, sacct and sinfo state before user retries or administrative changes replace it.
- 2
Join application evidence
Attach first-rank CUDA, NCCL, DeepSpeed or framework errors to the Slurm job and step.
- 3
Run an allocation-shaped control
Test the same resource shape, launcher and container on a known-good node or partition.
- 4
Close with a reason
Record whether the correction belonged to the request, environment, software stack, fabric or hardware.
What Denpex contributes
- Slurm job and GPU error encyclopedia coverage with official scheduler references
- Scheduler hooks, job wrapping and local Prometheus output
- Air-gapped deterministic diagnosis for environments without outbound access
- NCCL, CUDA, DeepSpeed and hardware context attached to scheduler incidents
Measure this in an evaluation
Use your incidents and timestamps. These are measurement definitions, not Denpex customer-outcome claims.
| Metric | Measurement |
|---|---|
| User ticket resolution time | Time from submitted scheduler evidence to a verified request, platform or application owner. |
| Repeat submission reduction | Rejected or pending jobs corrected before users resubmit the same impossible resource shape. |
| Node return-to-service time | Time from drain to a passed health control, excluding blind resumes that later fail again. |
| GPU allocation loss | Measured GPU hours consumed by failed jobs and preventable queue or node-state errors. |
Start with the technical evidence
Slurm GPU error reference
Decode GRES, QOS, cgroup, pending, drained-node and time-limit messages.
NCCL environment generator
Build and inspect environment settings for on-prem, Slurm and framework stacks.
Topology and fabric preflight
Check GPU to NIC affinity, PCIe topology and measured collective performance.
Self-hosted deployment
Run in-VPC or air-gapped diagnosis with local monitoring output.
Frequently asked questions
Can Denpex run without internet access on an HPC cluster?
Yes. The documented local mode uses the bundled deterministic engine and pattern database on the host, stores incident records locally and exposes Prometheus metrics without an API key. Live research and the hosted fallback are unavailable in that mode.
Does Denpex need Slurm administrator privileges?
Basic job wrapping and saved-log diagnosis do not require root. Scheduler hooks and node-level telemetry should be installed through the center's normal administrative review, with only the permissions required for the selected evidence path.
How does Denpex distinguish host OOM from CUDA OOM?
The Slurm cgroup event, sacct MaxRSS and host kernel log identify host-memory enforcement. A CUDA OOM comes from the framework and device allocator. Keeping both timelines attached to the job prevents one memory boundary from being mistaken for the other.
Can it support multiple research frameworks?
The corpus and deterministic engine cover scheduler, CUDA, NCCL and framework-specific signatures, including PyTorch distributed, DeepSpeed, vLLM and JAX families. The evaluation should include the center's actual modules, containers and launcher conventions.
Prove it on incidents your team already resolved
Freeze the expected owner and outcome, replay the evidence, then compare operator time and decision quality with your current process.
Build the evaluation plan