Tensor device mismatch
Application bugRuntimeError: Expected all tensors to be on the same deviceAn operator received operands on CPU and CUDA, or across different CUDA device indices, in a multi-GPU or model-parallel step.
First action: Run with CUDA_LAUNCH_BLOCKING=1 and print tensor devices via torch.cuda.current_device() right before the failing operation to verify every tensor and registered buffer moved to the same rank.
Full entry: root cause, fix and prevention