Skip to content
Troubleshooting

Troubleshooting

Step-by-step fixes for common issues. Each entry includes the symptom, root cause, and a verified resolution path. If you can't find your problem here, email support@denpex.com.

Agent: 'denpex.py: command not found'

Symptom

Running denpex.py produces a 'command not found' error even after installation.

Root cause

The file is not executable or the directory containing it is not on your system PATH.

Resolution steps

  1. Run chmod +x denpex.py to make the file executable.
  2. Alternatively, invoke it directly with Python: python denpex.py run -- your_training_command.
  3. If you installed via pip, verify the installation path is on PATH: python -m denpex.py run -- your_training_command.
  4. On SLURM clusters, add the denpex installation directory to PATH in your sbatch script before calling denpex.py.

Verify the fix

Run python denpex.py --version. You should see the installed version number.

Agent: 'Authentication failed: invalid API key'

Symptom

The agent starts but immediately fails with an authentication error.

Root cause

The DENPEX_API_KEY environment variable is not set, is set incorrectly, or the key has been deleted or rotated.

Resolution steps

  1. Go to Dashboard → Monitoring → API Keys and verify your key exists and is active.
  2. Copy the key exactly (no trailing spaces or newlines).
  3. Set it in your environment: export DENPEX_API_KEY=dpx_your_key_here.
  4. Verify it is set: echo $DENPEX_API_KEY.
  5. If running on SLURM, add it to your sbatch script with #SBATCH --export=DENPEX_API_KEY=dpx_... or use srun --export.
  6. If using Docker or Kubernetes, pass it as an environment variable in your container spec.

Verify the fix

Run python denpex.py auth test. It should print 'Authentication successful.'

Agent: heartbeats accepted but no diagnosis appears after a crash

Symptom

The agent is running and sending heartbeats (visible in Dashboard → Monitoring), but after a training crash no diagnosis appears.

Root cause

The agent process is not the parent of the training command, so it never receives the exit signal when training crashes.

Resolution steps

  1. Ensure the agent wraps your training command using python denpex.py run -- your_training_command.
  2. Do not launch the training command as a background process from within your script. The agent must be the direct parent process.
  3. If your scheduler (SLURM, Kubernetes) launches the training command independently, use the attach mode: python denpex.py attach --pid $(pgrep -f your_training_script).
  4. Check that the agent has network access to api.denpex.com on port 443.
  5. Review agent logs at /tmp/denpex-agent.log for any error output.

Verify the fix

Intentionally crash a test run with a known failure (e.g. kill -9 the training process). A diagnosis should appear in the dashboard within 30 seconds.

Diagnosis returns 'No match: LLM fallback in under 60s'

Symptom

The deterministic diagnosis returns no match and falls back to the LLM path.

Root cause

The failure did not meet the confidence threshold against any of the 11,700+ documented failure patterns. This can indicate a novel failure, a cluster-specific variant, or log output that was truncated before the meaningful signal.

Resolution steps

  1. Wait for the LLM fallback result. It typically arrives within 60 seconds on Team plans.
  2. If LLM fallback is disabled (UNKNOWN response with no fix), enable it in Dashboard → Settings → Diagnosis → LLM Fallback.
  3. Review whether log collection is complete. If the training process was killed before writing its full stderr, the failure signal may be incomplete.
  4. If this failure class recurs, file it from the diagnosis view in the dashboard. The team will add a deterministic pattern for it.
  5. On Team plans and above, record the corrected fix with the thumbs-down flow so the outcome is captured against that signature.

Verify the fix

Check the diagnosis confidence score in the API response. Scores above 0.7 are high-confidence deterministic matches; below 0.5 indicates the LLM path was used.

Slack alert: 'invalid_auth' or alerts not delivering

Symptom

Slack alerts are not arriving, or the Denpex webhook test returns an 'invalid_auth' error.

Root cause

The Slack webhook URL is expired, the workspace revoked the webhook, or the webhook was created for a channel that no longer exists.

Resolution steps

  1. In your Slack workspace, go to Apps → Incoming Webhooks and check whether the Denpex webhook is listed as active.
  2. If it shows as revoked or missing, create a new webhook for your target channel.
  3. Copy the new webhook URL (starts with https://hooks.slack.com/services/) and update it in Dashboard → Alerts → Slack.
  4. Click Send test alert on the Slack channel row in Dashboard → Alerts. It posts a synthetic diagnosis to that channel so you can confirm delivery end to end.
  5. Ensure the Denpex app has not been removed from your Slack workspace. If it was removed, the webhook is permanently invalid.

Verify the fix

Click Send test alert on the Slack channel row (Dashboard → Alerts). You should receive a test diagnosis in the configured channel within 10 seconds.

Login: 'CSRF validation failed' or OAuth state mismatch

Symptom

Clicking 'Sign in with Google / GitHub / Discord' redirects back to the login page with a CSRF or state mismatch error.

Root cause

OAuth state mismatch, typically caused by a stale session cookie, a browser extension that modifies redirect URLs, or navigating to the OAuth callback from a different browser tab.

Resolution steps

  1. Clear cookies for denpex.com: go to your browser settings, find cookies for denpex.com, and delete them all.
  2. Try the OAuth flow in an incognito or private browsing window to rule out extension interference.
  3. Disable any browser extensions that modify headers or redirect URLs (ad blockers, privacy extensions) and retry.
  4. Ensure you are completing the OAuth flow in the same browser tab that initiated it. Do not open the callback URL in a new tab.
  5. If the error persists, try a different browser.

Verify the fix

A successful OAuth login redirects to the dashboard with your account visible in the top-right corner.

High diagnosis latency (more than 30 seconds)

Symptom

A diagnosis is taking longer than expected or stops before returning a result.

Root cause

The diagnosis engine runs in tiers: deterministic pattern matching (fastest) → community research (GitHub/Stack Overflow) → LLM fallback (slowest). Latency increases for failures that require later tiers. Network latency to api.denpex.com can also contribute.

Resolution steps

  1. Check the diagnosis tier in the response. If it returned via LLM fallback, 30 to 60 seconds is expected.
  2. If community research (GitHub/SO) is being triggered frequently, send the failing logs in from the diagnosis view. Recurring classes get a deterministic pattern, which is the tier that short-circuits the slow path.
  3. For latency-critical deployments, enable the in-VPC agent (Scale+ plan) which runs the deterministic pattern engine locally with no network round-trip for the failure classes in its on-device corpus.
  4. Check your network path to api.denpex.com. High latency from your cluster to the API adds directly to diagnosis time.

Verify the fix

Check the x-denpex-diagnosis-tier header in the API response. Deterministic matches should be quick; the deep path can use most of the API's 290-second ceiling while it tests novel or ambiguous failures.

Stripe checkout: 'Billing not configured' or payment page not loading

Symptom

Clicking 'Upgrade' or 'Subscribe' produces a billing error or the Stripe checkout page does not load.

Root cause

A transient issue with our Stripe configuration. This is a server-side error, not a problem with your payment method.

Resolution steps

  1. Refresh the page and try again. Transient Stripe errors often resolve on retry.
  2. If the error persists for more than a few minutes, email support@denpex.com with the error message and the time you saw it.
  3. Do not enter payment information into a page showing an error. Wait for a clean checkout page.

Verify the fix

A working checkout page loads the Stripe payment form with the plan name and price visible in the order summary.

Logs being shipped on a paid plan: privacy concern

Symptom

Logs appear to be leaving your infrastructure even though you expected data to stay local.

Root cause

On Free and Team plans, logs are processed in the Denpex cloud API. Raw log lines are deleted immediately after diagnosis; only anonymized diagnostic metadata is retained for cross-run comparison. To prevent any egress, you need the in-VPC agent.

Resolution steps

  1. To prevent all log egress: upgrade to Scale+ and enable the in-VPC agent (docs at /docs/agent/self-hosted).
  2. The in-VPC agent runs the full pattern engine locally. Only diagnosis metadata (failure class, severity, rank ID) is sent to the API, no raw log content.
  3. Set DENPEX_LOCAL=1 in your environment to enable zero-egress mode on the Scale+ plan.
  4. To verify no raw logs are leaving: run Wireshark or tcpdump on the agent host and confirm outbound payloads contain only structured JSON metadata, not raw log lines.

Verify the fix

With DENPEX_LOCAL=1 set, the agent log at /tmp/denpex-agent.log will print 'Running in zero-egress local mode' at startup.

Still stuck?

If your issue isn't listed above, open the diagnosis console and paste your error output. You'll get a structured diagnosis in seconds. For account and billing issues, email support@denpex.com.

Open diagnosis console