Frequently asked questions

You have two options. (1) Paste logs into the web console, no install. (2) Install the single-file Python agent, which wraps your training command and ships the last 500 log lines on a non-zero exit. We never install a daemon or modify the training process.

No. The agent is completely passive during training. It only activates on job failure. Zero overhead on forward / backward passes, zero overhead during normal operation.

Yes. Paste logs from any past failure into the console. T0 (anonymous or negatively classified): 3/day. T1 (identified): 10/day. T2 (current verified organization proof): 50/day. T3 (active trial): the persisted trial cap, normally 50/day. T4 (paid): unlimited.

In distributed PyTorch collective operations like all-reduce, every participating GPU must reach the barrier to complete the step. If a worker on another rank silently dies from an unhandled CUDA out of memory error, segfault, or network drop, Rank 0 waits at the barrier until the default watchdog timer expires (typically 1,800 seconds) and raises a timeout. The true root cause occurred earlier on the silent rank, but Rank 0 is the one that logs the fatal timeout exception.

Instead of inspecting single-node logs in isolation, Denpex reconciles log streams across all participating ranks onto a single synchronized timeline. It identifies the first rank to diverge before the timeout cascade, cross-references CUDA driver states and host kernel events (including dmesg OOM kills and NVIDIA Xid errors), and determines whether the failure stemmed from activation sizing, imbalanced batches, or allocator fragmentation.

NVIDIA DCGM monitors device-level hardware health, and Prometheus records metric time series like temperature and memory utilization. Neither tool understands distributed PyTorch rank topologies or collective dependency graphs. When a multi-node job deadlocks, metrics show flatline utilization across all nodes. Denpex analyzes the actual execution logs, identifies which rank failed first, and provides the exact code, environment variable, or configuration change needed to fix it.

No. The agent masks PII/PHI client-side before egress by default; in strict mode it ships only anonymized failure signatures, so raw logs never leave. Raw logs are processed in memory and never written to durable storage. For zero egress, DENPEX_LOCAL=1 runs the deterministic engine entirely on-host. Nothing leaves the cluster at all.

If no pattern match exists with ≥60% confidence, Denpex falls back to an LLM-inferred diagnosis, clearly marked. You always know the confidence level of your diagnosis.

Yes. Spot preemption is classified separately from software failures. Denpex distinguishes between PREEMPTION events and actual training failures, so your metrics aren't polluted by cloud provider interruptions.

Partially today, natively soon. The console and the AI fallback already accept JAX/Flax stack traces and XLA compiler errors, paste them like any other log and you'll get a diagnosis with a confidence level. Deterministic JAX/Flax pattern packs (pjit/sharding mismatches, XLA OOM, TPU runtime faults, host-callback deadlocks) and TPU-native telemetry are in progress on the public roadmap. If you're a university or national-lab team on JAX, tell us at /contact. JAX users get priority access to the packs as they land.

Confirming a fix records it in your team's knowledge base against its failure class, scoped to your organisation. The next diagnosis of that class, for anyone on your team, leads with your confirmed fix and keeps the engine's recommendation underneath. Your corrected wording wins over the engine's if you edited it. Browse and search at /knowledge-base; the API is GET /api/kb/entries.

Yes, by giving it both runs. Paste the artifacts from the last good run and the failing one at /diff (free, no account) or POST them to /api/diagnose/env-diff on Team and above. Driver, CUDA, NCCL, framework, packages and NCCL_* variables are diffed and each change is ranked by how strongly it correlates with the failure class. Fetching your last good run automatically is in build.

Yes. DENPEX_LOCAL=1 runs the canonical deterministic pipeline entirely on-host with zero egress: patterns, causal detectors, Xid routing, cross-rank cascade analysis, and encyclopedia retrieval. The signed bundle uses a Python 3.8+ bridge plus a prebuilt Node.js 18+ engine; it needs no server, internet connection, API key, pip package, or npm install. Only the hosted LLM fallback and live research are unavailable offline. White-label / OEM is available on Data Center.

Yes. T0 (anonymous or negatively classified): 3/day. T1 (identified): 10/day. T2 (current verified organization proof): 50/day. T3 (active trial): the persisted trial cap, normally 50/day. T4 (paid): unlimited. A verified work mailbox, current disposable-domain checks, and independent company evidence can activate one 30-day Team trial per company, with up to 50 diagnoses a day. No particular mail provider is required. If we cannot verify the organization automatically, your existing diagnosis allowance remains available and support can review the domain. No card is required, and the trial does not become a subscription automatically.

The initial base subscription fee may be refunded within 7 calendar days for an unresolved technical issue. Consumed metered overage is non-refundable. Annual subscriptions may be cancelled with 30 days' notice, with any refund pro-rated for the unused base subscription only. Read the refund policy.