Framework-native integration

Install

The integration module is a single stdlib-only file (no dependencies beyond the Python standard library). Drop it next to your training script:

# from the repo
cp agent/denpex_integrations.py /your/project/

# or just download the single file
curl -O https://denpex.com/agent/denpex_integrations.py

# set your API key (the same dpx_... key the agent uses)
export DENPEX_API_KEY=dpx_your_key_here

1. Context manager (any framework)

Wraps any training block. On an uncaught exception, ships the captured stdout/stderr and prints the diagnosis. Works with plain PyTorch, JAX, Ray, anything.

import denpex_integrations as denpex

with denpex.watch(job_name="llama3-finetune"):
    model.train()
    for batch in dataloader:
        loss = model(batch)
        loss.backward()
        optimizer.step()

2. PyTorch Lightning callback

Pass it as a trainer callback. Captures output for the run and ships logs on the on_exception hook.

import lightning as L
from denpex_integrations import DenpexLightningCallback

trainer = L.Trainer(
    max_epochs=100,
    callbacks=[DenpexLightningCallback(job_name="llama3-ltn")],
)
trainer.fit(model, datamodule=dm)

3. HuggingFace Trainer callback

Add it to a HuggingFace Trainer. It hooks on_train_begin to start capture and watches for NaN/Inf in the log values. If transformers is installed, the callback subclasses TrainerCallback so it registers cleanly; otherwise it works via duck-typing.

from transformers import Trainer, TrainingArguments
from denpex_integrations import DenpexTrainerCallback

trainer = Trainer(
    model=model,
    args=TrainingArguments(output_dir="./out"),
    train_dataset=ds,
)
trainer.add_callback(DenpexTrainerCallback(job_name="bert-ft"))
trainer.train()

4. Experiment trackers — W&B, MLflow, TensorBoard

Attach a diagnosis straight to a failed run. In an except around your loop, Denpex reads the run's id/URL and recent metric history and diagnoses the failure — flagging NaN / divergence even when only metrics are available.

import denpex_integrations as denpex

try:
    trainer.train()
except Exception:
    denpex.push_wandb_run()                  # reads wandb.run + recent history
    # denpex.push_mlflow_run()               # active MLflow run + metrics
    # denpex.push_tensorboard("runs/exp1")   # latest tfevents scalars
    raise

Prefer a webhook? Point a W&B Automation (or any tracker alert) at the ingest URL — the run context is included, and a metrics summary makes the failure diagnosable:

curl -X POST https://api.denpex.com/api/integrations/ingest \
  -H "X-Denpex-Key: $DENPEX_API_KEY" -H "Content-Type: application/json" \
  -d '{"provider":"wandb",
       "run":{"name":"llama3-ft","url":"https://wandb.ai/acme/p/runs/abc"},
       "logs":"CUDA out of memory on rank 0"}'

The diagnosis is stored in your history (linked to the run) and routed to your Slack / PagerDuty channels. Team plan and up.

5. Live monitor — catch divergence in seconds

The callbacks above are reactive — they ship after a crash. The live monitor is proactive: feed it your metrics each step and it flags NaN, a gradient-norm spike, loss divergence, or a throughput collapse the moment it starts — and can auto-checkpoint so you resume from the last good state instead of hitting the NCCL watchdog hours later.

from denpex_live import LiveMonitor

mon = LiveMonitor(job_name="llama3-ft",
                  on_checkpoint=lambda: trainer.save_checkpoint("denpex-rescue.ckpt"))

for step, batch in enumerate(loader):
    loss = train_step(batch)
    mon.update(step, {"loss": loss.item(), "grad_norm": gn, "tokens_per_sec": tps})

PyTorch Lightning users can just add the callback:

from denpex_live import DenpexLiveCallback

trainer = L.Trainer(callbacks=[DenpexLiveCallback(job_name="llama3-ft", auto_checkpoint=True)])

6. MCP server — diagnose from Claude, Cursor, or any MCP client

Denpex ships a Model Context Protocol server, so you can diagnose a crash without leaving your AI tool. Ask Claude Desktop, Cursor, or Claude Code to “diagnose the log at ./logs/rank0.err” and it reads the file off disk and hands back the root cause and the fix. An API key is optional — diagnose_crash, lookup_failure, and analyze_cascade work fully offline; a dpx_… key unlocks the cloud LLM fallback on novel errors, your diagnosis history, quota, and fleet incidents.

Add it to your client config (claude_desktop_config.json, .cursor/mcp.json, or the equivalent for your client):

{
  "mcpServers": {
    "denpex": {
      "command": "npx",
      "args": ["-y", "denpex-mcp"],
      "env": { "DENPEX_API_KEY": "dpx_your_key_here" }
    }
  }
}

Claude Code users can add it in one line:

claude mcp add denpex -e DENPEX_API_KEY=dpx_your_key_here -- npx -y denpex-mcp

For air-gapped clusters, set DENPEX_LOCAL=1 to diagnose entirely on-host — the bundled pattern engine and failure encyclopedia run locally and nothing leaves the machine. Available on Growth and Data Center plans.

7. GitHub Action — Auto-Diagnose Failed CI/CD Runs

Engineers hate digging through 50,000 lines of GitHub Actions logs when a training run fails. The Denpex GitHub Action automatically detects when your workflow fails, grabs the logs, and comments on your PR with the exact root cause and suggested fix.

# .github/workflows/train.yml
name: Train

jobs:
  train:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python train.py
      
      - name: Diagnose failure
        if: failure()
        uses: denpex/github-action@v1
        with:
          api-key: ${{ secrets.DENPEX_API_KEY }}

100% Private: Denpex never clones your source code. The action only uploads the workflow logs necessary to diagnose the failure. Your IP remains completely secure.

8. Kubernetes DaemonSet — Zero-Touch Enterprise Observability

The Holy Grail for ML Platform Engineers running 10,000-GPU clusters. Instead of asking researchers to modify their code, deploy the Denpex DaemonSet once.

kubectl apply -f https://raw.githubusercontent.com/MisaMisaAI/denpex-action/main/daemonset.yaml

The daemon runs silently in the background on every node, watching /var/log/syslog, dmesg, and pod logs. When a researcher's standard torchrun job crashes, the daemon instantly catches the CUDA OOM or Xid fault, correlates the hardware state, and posts the diagnosis to your Slack channel automatically.

What gets captured

• A bounded ring buffer of the last 4,000 lines of stdout+stderr — flat memory for long runs, and the crash is always at the tail.
• The full traceback on an uncaught exception.
• The measured runtime (job start → crash), so the product attaches a real GPU-hours/$ figure to the failure, not a hypothetical one.
• Output passes through to your console unchanged — capture never silences your logs.

Cost intelligence

Because the callbacks ship the measured runtime, every diagnosed failure from a callback gets a real cost figure: “this failure burned ~340 GPU-hours (~$1,020) before it was caught.” These roll up on the dashboard into “Denpex surfaced N GPU-hours of waste this month” — the number a data-center buyer uses to justify the spend.

Zero dependencies

Like the CLI agent, denpex_integrations.py uses only the Python standard library (urllib,atexit, sys). Lightning and Transformers are imported lazily — the file loads and the context manager works even if neither framework is installed. This keeps the install footprint at zero, which matters inside locked-down training environments.