Skip to content

Operator task guide

Diagnose QLoRA setup failures before a full training run

Start with the first exception from the same Python environment and GPU as the failing trainer. A CUDA visibility failure, a missing bitsandbytes library, an unsupported 4-bit kernel and an adapter configuration error require different checks.

Verify four stages independently: CUDA visibility, 4-bit model loading, the first forward and backward pass, and an adapter optimizer step. A model that loads successfully can still fail at the next stage.

Reviewed . Reference guidance is not a diagnosis of your workload.

Step-by-step method

  1. 1

    Check CUDA and the selected packages

    Run these commands inside the failing container. If PyTorch cannot see the intended GPU, investigate assignment, driver visibility and the installed PyTorch build before changing quantization.

    python -m pip show torch bitsandbytes transformers peft
    python -c "import torch; print(torch.cuda.is_available(), torch.version.cuda)"
    python -m bitsandbytes
  2. 2

    Separate the loader from the kernel

    A missing shared library identifies a loader problem. A no-kernel-image error during a 4-bit operation points to a different check: compare the selected wheel, GPU compute capability and documented support for the package release.

  3. 3

    Verify a real adapter step

    Load a known supported model with the intended 4-bit configuration, prepare it for k-bit training and confirm PEFT selected the expected trainable adapter parameters. Run a small nonprivate batch through forward, backward and one optimizer step. Check finite loss and gradients and confirm an intended adapter parameter changed.

  4. 4

    Reintroduce the production training configuration

    Add the original model, sequence length, batch size and distributed launch one at a time. Keep the first error and the failing stage, especially if the reduced example passed.

Where the QLoRA job first fails

Signals, meanings and actions for diagnose qlora setup failures before a full training run.
SignalWhat it meansNext action
CUDA is unavailable in PyTorchThe trainer cannot use the intended NVIDIA GPU.Check GPU assignment and the installed PyTorch build in the failing environment.
bitsandbytes cannot load a native libraryThe selected backend is unavailable or one of its dependencies failed.Read the first loader error and compare the installed release with its support matrix.
No kernel image during matmul_4bitThe selected binary may lack a compatible kernel for that GPU.Compare compute capability and wheel support before changing model settings.
Load passes but backward or update failsThe failure is later than model loading.Check the PEFT configuration, trainable parameters, first traceback and the exact batch.

Evidence checklist

  • Full first exception and stage
  • GPU model and compute capability
  • PyTorch, bitsandbytes, Transformers and PEFT versions
  • Quantization settings and trainable adapter targets

Common mistakes

Treating successful import as a GPU test

Import can succeed before the first 4-bit operation exercises the selected kernel.

Changing several package versions at once

Keep the baseline and change one compatibility variable so the next result is explainable.

Frequently asked questions

Does a successful model load prove QLoRA is working?

No. Exercise forward, backward and an adapter optimizer step. Check finite values and verify an intended adapter parameter updates.

Apply the method to your incident

Use the three free diagnoses to review your error and relevant evidence. Keep reference guidance separate from the cause and recovery status of your own workload.

Diagnose your incident