← Back to reference

Silent Corruption Detection Limits

What We Can Detect

The denpex SDK ships a canary SDC runner that:

  • Runs a small fixed-input microbatch on a sampled rank in parallel with the real microbatch.
  • Compares outputs bit-for-bit using torch.allclose with a 1-ULP tolerance.
  • Detects layer-level checksums via XXH3.
  • Validates Adam moments across ranks at the same step.

What We CANNOT Detect

  • Bit flips masked by FP32 rounding. A single-bit flip that happens to round to the same value passes the canary.
  • Silent drift in long-tailed tensors. A 1 % shift in a rare-occurrence slice may not exceed the 1-ULP tolerance.
  • Correlated hardware faults. A clock that runs 0.1 % slow produces identical outputs.
  • Driver-level reordering. If the kernel reorders reductions, our hooks may not see the original ordering.

Empirical Coverage

From the synthetic corpus:

  • Single-bit ECC failure in HBM: detected 91 % of the time at 2 GB/s DRAM.
  • Cumulative drift (10 + ULP): detected 99 % of the time over a 1000-step window.
  • Mid-stream compiler reordering: not detected; we recommend the user turn on strict determinism via PyTorch's --deterministic-algorithms.

Recommendations to Customers

  1. Enable --deterministic-algorithms for any reproducibility- critical run.
  2. Enable the canary with a 1-in-10 sampling rate.
  3. Run the SDC-test corpus in CI before any infra change.
  4. Carry manifest plus smoke-inference as part of every checkpoint.

Limits Are Spelled Out Here

Not in marketing. Reproduce from docs/benchmarks/accuracy/corpus.csv and the script docs/benchmarks/silent-corruption/run.sh.