Data Pipeline error references

Browse 55 published references for data pipeline failures in GPU training and inference. Match the message and context from your run, then open the reference for possible causes, evidence checks and sources. A matching message alone does not confirm the cause.

• Arrow IPC Format Error

  Apache Arrow IPC errors occur when serialized Arrow data is corrupted or has version incompatibilities.

• Audio Channel Mismatch

  Audio channel mismatches (mono vs stereo) cause model errors or poor performance in audio ML pipelines.

• Audio Data Loading Error

  Audio data loading errors occur when audio files are corrupted, format unsupported, or sample rates mismatch.

• Audio Sample Rate Mismatch

  Audio sample rate mismatches between training data and pretrained models cause poor ASR/TTS performance.

• Augmentation Pipeline Error

  Augmentation pipeline errors cause inconsistent or corrupted training data when transforms fail or produce unexpected outputs.

• Augmentations Too Aggressive

  Overly aggressive data augmentations hurt model performance by distorting critical features in training data.

• BatchNorm DDP Synchronization Issue

  BatchNorm synchronization issues in DDP cause inconsistent statistics between ranks and degraded model quality.

• Class Imbalance

  Class imbalance causes models to predict majority class and have poor performance on minority classes.

• Contrastive Learning Augmentation

  Contrastive learning augmentation pipelines must be carefully tuned to create useful positive pairs without destroying semantic content.

• cuFileRead failed with error 5002 CU_FILE_DRIVER_NOT_INITIALIZED

  GPUDirect Storage was requested but the cuFile driver is not loaded, so the nvidia-fs kernel module is unavailable. Reads either fail outright or silently fall back through host memory, losing the entire benefit of GDS. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent storage failures.

• Data Augmentation Error

  Data augmentation errors occur when augmentation libraries fail or produce corrupted outputs.

• Data Pipeline / DataLoader Stalls

  Training is bottlenecked or completely halted due to the DataLoader failing to fetch the next batch in time.

• Data Versioning Issue

  Data versioning issues occur when training data changes between runs without tracking, making results non-reproducible.

• Dataloader num_workers=0 Too Slow

  DataLoader with num_workers=0 is often too slow for GPU training, causing the GPU to wait for data.

• Dataloader Worker Failure

  Dataloader worker failures stall training or produce corrupted batches.

• DataLoader worker is killed by signal: Bus error shared memory exhausted

  PyTorch DataLoader uses shared memory (/dev/shm) to pass tensors to the main process. K8s defaults /dev/shm to 64MB. When data is large, this fills up rapidly, leading to Bus error or Pod OOMKilled. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent storage failures.

• Dataset Bias

  Dataset bias causes models to learn spurious correlations that don't generalize to real-world data.

• Determinism Broken

  Broken determinism causes non-reproducible training runs, making debugging and comparison difficult.

• HDF5 Concurrent Read Deadlock with Multiprocessing

  The HDF5 library maintains internal state and file locks. It is not safe to fork a process that has an open HDF5 file handle. When PyTorch creates child workers via fork, they all inherit the same file descriptor and internal HDF5 state, causing corruption or deadlocks when they attempt to read concurrently.

• HDF5 Data Corruption

  HDF5 file corruption causes training to fail with I/O errors or silently load incorrect data.

• HuggingFace datasets .map() 'subprocess abruptly died' on Windows

  Dataset preprocessing with datasets.map(num_proc>1) crashes on Windows with 'One of the subprocesses has abruptly died during map operation'. Windows uses spawn rather than fork, so non-picklable map functions or native child crashes kill the worker; the reliable fix is single-process map or WSL2.

• HuggingFace Tokenizer Error

  HuggingFace tokenizer errors crash training when tokenizer configuration mismatches model architecture.

• Image Corruption Detection

  Image corruption detection helps identify and skip/fix corrupted images that would otherwise crash training.

• Image Resize Artifacts

  Image resize artifacts degrade model accuracy when resize method (BILINEAR, BICUBIC, LANCZOS) doesn't match pretrained model expectations.

• ImageNet Preprocessing Mismatch

  ImageNet preprocessing mismatches cause pretrained models to underperform because input normalization differs from training.

• LLM Tokenization Truncation

  LLM tokenization truncation silently cuts long sequences, causing loss of information in long documents.

• LMDB Corruption

  LMDB corruption occurs when LMDB files are not properly closed, transactions interrupted, or map_size exceeded.

• Missing Data Augmentation

  Missing data augmentation causes overfitting and poor generalization, especially on small datasets.

• MMap File Handle Exhaustion

  Memory-mapped (mmap) file handles are exhausted when datasets have many large files, causing data loading failures.

• Multi-Label Class Imbalance

  Multi-label class imbalance causes poor performance on rare labels and biased predictions toward frequent labels.

• Multi-Task Learning Conflict

  Multi-task learning conflicts arise when tasks have different scales, gradients, or learning dynamics that destabilize training.

• NCCL Watchdog Timeout due to Uneven Dataset Sharding

  When sharding datasets across multiple workers, if the total number of samples is not perfectly divisible by the world size and drop_last=False, some ranks will have fewer batches. These ranks will finish their loop early, while the remaining ranks enter a collective operation (e.g., gradient all_reduce) expecting participation from all ranks, leading to a hang until the watchdog times out.

• nfs: server not responding still trying task blocked for more than 120 seconds

  D-State Task Hang - A process is stuck in uninterruptible sleep (D-state), waiting on hardware or network I/O. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent storage failures.

• Parquet Schema Mismatch

  Parquet schema mismatches occur when reading parquet files with different schemas than expected by the loading code.

• Pinned Memory Transfer Issue

  Pinned memory transfer issues cause slow GPU-CPU data transfer or OOM errors.

• Python Multiprocessing Fork Issue

  Python multiprocessing fork issues cause workers to deadlock or share state incorrectly across ranks.

• PyTorch Lightning Fault-Tolerant Resume Fails with DataLoader num_workers>1

  Resuming a run with PyTorch Lightning's experimental fault-tolerant training raises a MisconfigurationException about the worker state when the DataLoader uses num_workers>1. The feature never reliably reconstructed per-worker iterator state and was removed in Lightning 2.0, so the only durable fix is a custom resumable sampler.

• PyTorch Multiprocessing Error

  PyTorch multiprocessing errors prevent DataLoader workers from functioning correctly.

• RAG Embedding Mismatch

  RAG embedding mismatches cause retrieval to return irrelevant documents when embedding model differs between indexing and query.

• Shuffle Buffer Too Small

  Small shuffle buffers cause poor data ordering during training, hurting model generalization.

• Streaming Data Error

  Streaming data errors occur when loading data from cloud storage or remote endpoints fails mid-training.

• Text Encoding Mismatch

  Text encoding mismatches (UTF-8 vs Latin-1) cause data loading failures or corruption in NLP training.

• TFRecord / tf.data Error

  TFRecord and tf.data errors occur when TensorFlow data pipeline has issues with the training data or pipeline configuration.

• TFRecord Corrupted Shard

  TFRecord corrupted shard errors occur when individual TFRecord files are truncated or have inconsistent examples.

• Tokenization Error

  Tokenization errors occur when text data has issues that prevent proper tokenization.

• Tokenizer Padding and Truncation Error

  Tokenizer padding and truncation errors cause sequence length mismatches and OOM during training.

• Tokenizer Padding Mismatch

  Tokenizer padding mismatches cause data loader errors or poor model performance when padding side or token ID differs.

• Tokenizer Version Mismatch

  Tokenizer version mismatches between training and inference cause different token IDs for same text, breaking model behavior.

• TorchData Pipeline Error

  TorchData (torch.utils.data.datapipes) errors occur when complex data pipelines have configuration or compatibility issues.

• Unsloth SFTTrainer: 'int' object has no attribute 'mask_token'

  Constructing an Unsloth SFTTrainer (often in Colab) raises AttributeError: 'int' object has no attribute 'mask_token'. The crash comes from API drift between Unsloth's patched trainer __init__ and a newer TRL/transformers SFTTrainer signature, where a positional integer argument lands where a tokenizer was expected.

• Validation Set Leakage

  Validation set leakage causes overly optimistic metrics because training data includes validation samples.

• Video Codec Mismatch

  Video codec mismatches between training data and pretrained video models cause loading failures or poor performance.

• WebDataset Error

  WebDataset errors occur when streaming dataset format has corrupted shards or URL issues.

• WebDataset TAR Corruption

  WebDataset TAR file corruption causes data loading failures and missing samples during training.

• WebDataset/TFRecord Decoding Crash Mid-Epoch

  A large tarball or TFRecord file was partially downloaded, truncated during transfer, or concurrently written to while the training job was reading it. The dataloader hits the truncated EOF mid-stream, throwing a fatal exception that kills the worker.