Skip to content
Home/Failures/Indices Device Mismatch

Fix: Indices Should Be on CPU or Same Device in PyTorch

Comprehensive guide to resolving PyTorch device placement faults during tensor indexing, embedding lookups, and gradient calculations.

Bottom Line Up Front (BLUF)

When PyTorch throws RuntimeError: indices should be either on cpu or on the same device as the indexed tensor, it means your indexing tensor or target tensor reside on different hardware devices, such as CPU versus CUDA.

# Immediate one-line fix:

indices = indices.to(target_tensor.device, non_blocking=True)

Try Denpex for 30 days. FREE.

No credit card. No automatic subscription. See whether Denpex helps with an incident you already understand.

Start your free 30-day Team evaluation

Up to 50 diagnoses a day for 30 days. Verify your work email to activate an eligible company evaluation, then replay a resolved incident and compare the evidence and next steps with your team's confirmed cause.

Use your company email (e.g. alex@yourcompany.com). Eligible company domains can activate a free 30-day Team evaluation (up to 50 diagnoses/day) after email and domain checks, regardless of mail host. Uncertain domains can request review from support@denpex.com. Free local diagnoses remain available without an account.

Three Deep-Dive Root Cause Scenarios

1. DataLoader Target and Label Tensors on CPU

Standard PyTorch DataLoaders return batches on CPU by default. When these batch labels or input token IDs are fed directly into embedding layers or loss functions residing on GPU without explicit device transfer, the runtime rejects the operation.

Bad Code (Crashes)

for input_ids, labels in dataloader:
    # input_ids moved, but labels stay on CPU
    outputs = model(input_ids.to('cuda'))
    loss = criterion(outputs, labels) 
    # RuntimeError: labels on cpu

Good Code (Correct)

device = next(model.parameters()).device
for input_ids, labels in dataloader:
    input_ids = input_ids.to(device, non_blocking=True)
    labels = labels.to(device, non_blocking=True)
    outputs = model(input_ids)
    loss = criterion(outputs, labels)

2. Advanced Index Slicing and Boolean Masking

Operations such as torch.gather, torch.index_select, and dynamic boolean masks generated via tensor comparisons often evaluate on CPU or default device context, failing when applied to GPU weights.

Bad Code (Crashes)

weights = torch.randn(100, 512, device='cuda')
# torch.where creates CPU tensor mask by default in some contexts
mask = (weights > 0) 
selected = torch.masked_select(weights, mask)

Good Code (Correct)

weights = torch.randn(100, 512, device='cuda')
mask = (weights > 0).to(weights.device)
selected = torch.masked_select(weights, mask)

3. Multi-GPU DistributedDataParallel (DDP) Pitfalls

Hardcoding .to('cuda') forces all training processes onto cuda:0. Ranks 1 through 7 encounter device mismatches or NCCL deadlocks because tensors are not mapped to their assigned local devices.

Bad Code (Crashes)

# Hardcoded device string breaks DDP ranks > 0
tensor = tensor.to('cuda')
indices = indices.to('cuda')
# Fails when rank is assigned to cuda:2

Good Code (Correct)

# Retrieve local rank from environment or trainer
local_rank = int(os.environ.get('LOCAL_RANK', 0))
device = torch.device(f'cuda:{local_rank}')
tensor = tensor.to(device)
indices = indices.to(device, non_blocking=True)

Defensive PyTorch Best Practices

  • Unified Device Inspection: Always fetch the target tensor device dynamically using tensor.device rather than relying on global default settings.
  • Asynchronous Transfers: Use non_blocking=True when moving pinned host memory tensors to GPU devices to overlap computation and data transfer.
  • Hugging Face Transformers: Leverage device_map='auto' during model instantiation to automatically place submodules and weights across available accelerators without manual device management.

Related Tools and Guides

GPU Log Analyzer

Parse CUDA crash traces, memory allocation errors, and stack traces instantly.

VRAM Calculator

Estimate memory overhead for model weights, activations, and gradient buffers.

XID Error Reference

Lookup NVIDIA driver error codes and hardware failure signatures.

Open VSX Registry Guide

Publish and secure extension releases with verified registry guidelines.

Frequently Asked Questions

Why does PyTorch require indices and tensors on the same device?

PyTorch CUDA kernels execute directly on the GPU hardware. Indexing operations require parallel memory lookups across high speed device memory. Crossing the PCIe bus between CPU RAM and GPU VRAM during kernel indexing would introduce severe performance bottlenecks and memory synchronization faults.

Does non_blocking=True work with all tensors?

Non-blocking transfers require pinned host memory, which can be enabled in PyTorch DataLoaders by passing pin_memory=True.