Framework-native integration

Install

The integration module is a single stdlib-only file (no dependencies beyond the Python standard library). Drop it next to your training script:

# from the repo
cp agent/denpex_integrations.py /your/project/

# or just download the single file
curl -O https://denpex.com/agent/denpex_integrations.py

# set your API key (the same dpx_... key the agent uses)
export DENPEX_API_KEY=dpx_your_key_here

1. Context manager (any framework)

Wraps any training block. On an uncaught exception, ships the captured stdout/stderr and prints the diagnosis. Works with plain PyTorch, JAX, Ray, anything.

import denpex_integrations as denpex

with denpex.watch(job_name="llama3-finetune"):
    model.train()
    for batch in dataloader:
        loss = model(batch)
        loss.backward()
        optimizer.step()

2. PyTorch Lightning callback

Pass it as a trainer callback. Captures output for the run and ships logs on the on_exception hook.

import lightning as L
from denpex_integrations import DenpexLightningCallback

trainer = L.Trainer(
    max_epochs=100,
    callbacks=[DenpexLightningCallback(job_name="llama3-ltn")],
)
trainer.fit(model, datamodule=dm)

3. HuggingFace Trainer callback

Add it to a HuggingFace Trainer. It hooks on_train_begin to start capture and watches for NaN/Inf in the log values. If transformers is installed, the callback subclasses TrainerCallback so it registers cleanly; otherwise it works via duck-typing.

from transformers import Trainer, TrainingArguments
from denpex_integrations import DenpexTrainerCallback

trainer = Trainer(
    model=model,
    args=TrainingArguments(output_dir="./out"),
    train_dataset=ds,
)
trainer.add_callback(DenpexTrainerCallback(job_name="bert-ft"))
trainer.train()

4. Experiment trackers, W&B, MLflow, TensorBoard

Attach a diagnosis straight to a failed run. In an except around your loop, Denpex reads the run's id/URL and recent metric history and diagnoses the failure. Flagging NaN / divergence even when only metrics are available.

import denpex_integrations as denpex

try:
    trainer.train()
except Exception:
    denpex.push_wandb_run()                  # reads wandb.run + recent history
    # denpex.push_mlflow_run()               # active MLflow run + metrics
    # denpex.push_tensorboard("runs/exp1")   # latest tfevents scalars
    raise

Prefer a webhook? Point a W&B Automation (or any tracker alert) at the ingest URL. The run context is included, and a metrics summary makes the failure diagnosable:

curl -X POST https://api.denpex.com/api/integrations/ingest \
  -H "X-Denpex-Key: $DENPEX_API_KEY" -H "Content-Type: application/json" \
  -d '{"provider":"wandb",
       "run":{"name":"llama3-ft","url":"https://wandb.ai/acme/p/runs/abc"},
       "logs":"CUDA out of memory on rank 0"}'

The diagnosis is stored in your history (linked to the run) and routed to your Slack / PagerDuty channels. Team plan and up.

5. Live monitor. Catch divergence in seconds

The callbacks above are reactive. They ship after a crash. The live monitor is proactive: feed it your metrics each step and it flags NaN, a gradient-norm spike, loss divergence, or a throughput collapse the moment it starts. And can auto-checkpoint so you resume from the last good state instead of hitting the NCCL watchdog hours later.

from denpex_live import LiveMonitor

mon = LiveMonitor(job_name="llama3-ft",
                  on_checkpoint=lambda: trainer.save_checkpoint("denpex-rescue.ckpt"))

for step, batch in enumerate(loader):
    loss = train_step(batch)
    mon.update(step, {"loss": loss.item(), "grad_norm": gn, "tokens_per_sec": tps})

PyTorch Lightning users can just add the callback:

from denpex_live import DenpexLiveCallback

trainer = L.Trainer(callbacks=[DenpexLiveCallback(job_name="llama3-ft", auto_checkpoint=True)])

6. MCP server. Diagnose from Claude, Cursor, or any MCP client

Denpex ships a Model Context Protocol server, so you can diagnose a crash without leaving your AI tool. Ask Claude Desktop, Cursor, or Claude Code to “diagnose the log at ./logs/rank0.err” and it reads the file off disk and hands back the root cause and the fix. An API key is optional, diagnose_crash, lookup_failure, and analyze_cascade work fully offline; a dpx_… key unlocks the cloud LLM fallback on novel errors, your diagnosis history, quota, and fleet incidents.

Add it to your client config (claude_desktop_config.json, .cursor/mcp.json, or the equivalent for your client):

{
  "mcpServers": {
    "denpex": {
      "command": "npx",
      "args": ["-y", "denpex-mcp"],
      "env": { "DENPEX_API_KEY": "dpx_your_key_here" }
    }
  }
}

Claude Code users can add it in one line:

claude mcp add denpex -e DENPEX_API_KEY=dpx_your_key_here -- npx -y denpex-mcp

For air-gapped clusters, set DENPEX_LOCAL=1 to diagnose entirely on-host. The canonical deterministic engine, causal detectors, pattern registry, and failure encyclopedia run locally and nothing leaves the machine. Works with no API key; hosted LLM/live-research enrichment and account history are unavailable in local mode.

7. GitHub Action. Auto-Diagnose Failed CI/CD Runs

Engineers hate digging through 50,000 lines of GitHub Actions logs when a training run fails. The Denpex GitHub Action automatically detects when your workflow fails, grabs the logs, and comments on your PR with the exact root cause and suggested fix.

# .github/workflows/train.yml
name: Train

jobs:
  train:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python train.py
      
      - name: Diagnose failure
        if: failure()
        uses: MisaMisaAI/denpex-action@main
        with:
          api-key: ${{ secrets.DENPEX_API_KEY }}

100% Private: Denpex never clones your source code. The action only uploads the workflow logs necessary to diagnose the failure. Your IP remains completely secure.

8. Kubernetes DaemonSet, Zero-Touch Enterprise Observability

The Holy Grail for ML Platform Engineers running 10,000-GPU clusters. Instead of asking researchers to modify their code, deploy the Denpex DaemonSet once.

kubectl apply -f https://raw.githubusercontent.com/MisaMisaAI/denpex-action/main/daemonset.yaml

The daemon runs silently in the background on every node, watching /var/log/syslog, dmesg, and pod logs. When a researcher's standard torchrun job crashes, the daemon instantly catches the CUDA OOM or Xid fault, correlates the hardware state, and posts the diagnosis to your Slack channel automatically.

What gets captured

• A bounded ring buffer of the last 4,000 lines of stdout+stderr. Flat memory for long runs, and the crash is always at the tail.
• The full traceback on an uncaught exception.
• The measured runtime (job start → crash), so the product attaches a real GPU-hours/$ figure to the failure, not a hypothetical one.
• Output passes through to your console unchanged. Capture never silences your logs.

Cost intelligence

Because the callbacks ship the measured runtime, every diagnosed failure from a callback gets a real cost figure: “this failure burned ~340 GPU-hours (~$1,020) before it was caught.” These roll up on the dashboard into “Denpex surfaced N GPU-hours of waste this month”, the number a data-center buyer uses to justify the spend.

Zero dependencies

Like the CLI agent, denpex_integrations.py uses only the Python standard library (urllib,atexit, sys). Lightning and Transformers are imported lazily. The file loads and the context manager works even if neither framework is installed. This keeps the install footprint at zero, which matters inside locked-down training environments.