An out-of-tree GPU or storage kernel module stops building after a kernel upgrade

DKMS rebuilds out-of-tree modules against each new kernel, but it can only recompile the source it was given. When the kernel removes or renames an internal function the module calls, the rebuild fails with an implicit declaration error and the node comes back from its upgrade without GPUDirect, without its parallel filesystem client, or in some cases without a working driver at all.

A post-upgrade build failure in which DKMS recompiles an out-of-tree kernel module against a kernel whose internal API has changed, so the module fails to compile or fails to load and the capability it provided disappears without an explicit alert.

Linux deliberately keeps no stable in-kernel API, so anything living outside the tree must track it. DKMS exists to rebuild those modules automatically and does exactly that, which means it will rebuild source the kernel has outgrown and surface the disagreement as a compile error. The upgrade succeeds from the package manager's point of view because the package is still installed; only the module is missing.

• A node reboots after routine patching and loses a capability nothing in the change record touched
• The failure is a compiler error in a log nobody reads, not an alert
• The module built correctly on this machine last month against the previous kernel
• Rolling the kernel back restores service, which makes the vendor look blameless and the kernel look at fault

• GPUDirect Storage and RDMA stacks, where peer-memory modules bridge the GPU driver and the fabric
• Parallel filesystem clients built as out-of-tree modules, including BeeGFS with GPU Direct support
• NVIDIA driver DKMS packages on distributions that ship kernels faster than the driver supports them
• Any cluster where unattended upgrades can move the kernel without moving the drivers with it

Apply the following checklist to a small reproduction: each box below is a positive signal that you are looking at this exact failure rather than a sibling in the same taxonomy.

• ✓Open /var/lib/dkms/<module>/<version>/build/make.log and take the first error line; it names the symbol and the source file, which is the whole diagnosis.
• ✓Compare the running kernel against the versions the module's release notes claim support for. A kernel newer than anything the vendor lists is the answer.
• ✓Check whether the module is loaded at all. A module that failed to rebuild leaves the package installed, so package state is not evidence that the capability exists.

/var/lib/dkms/beegfs/8.1.0/build/build/../source/common/storage/RdmaInfo.c:58:13: error: implicit declaration of function 'ib_peer_memory_client_register'
modprobe: ERROR: could not insert 'nvidia_peermem': Invalid argument
dmesg: nvidia_peermem: Unknown symbol ib_register_peer_memory_client

Timestamps and exact values vary across runs, but the pattern. An info-level start, an early WARN, an ERROR carrying the symptom. Is the actual fingerprint you should alert on. The Denpex platform flags this combination automatically.

Moving to a module release built for the running kernel supplies source that calls functions which exist, so the rebuild completes and the capability returns. Correcting the install order for peer-memory components works on the same principle from the linking side: the driver can only bind to fabric symbols that were present when it built.

// Typical pattern:
import torch.optim as optim
optimizer = optim.AdamW(model.parameters(), lr=3e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_training_steps)
for step in range(num_training_steps):
    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()
    optimizer.step()
    scheduler.step()

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.