[ { "failure_id": "lf_github_issue_2873", "source_url": "https://github.com/meta-pytorch/torchtune/issues/2873", "source_type": "github_issue", "failure_type": "CHECKPOINT_CORRUPTION", "title": "Error running distributed Qwen3-32B lora", "error_message": "Traceback (most recent call last):\n[rank3]: File \"/home/allen.philip/src/finetune/torchtune/.venv/lib/python3.11/site-packages/recipes/lora_finetune_distributed.py\", line 908, in \n[rank3]: sys.exit(recipe_main())\n[rank3]: ^^^^^^^^^^^^^\n[rank3]: File \"/home/allen.philip/src/finetune/torchtune/.venv/lib/python3.11/site-packages/torchtune/config/_parse.py\", line 99, in wrapper\n[rank3]: sys.exit(recipe_main(conf))\n[rank3]: ^^^^^^^^^^^^^^^^^\n[rank3]: File \"/home/allen.philip/src/finetune/torchtune/.venv/lib/python3.11/site-packages/recipes/lora_finetune_distributed.py\", line 902, in recipe_main\n[rank3]: recipe.setup(cfg=cfg)\n[rank3]: File \"/home/allen.philip/src/finetune/torchtune/.venv/lib/python3.11/site-packages/recipes/lora_finetune_distributed.py\", line 288, in setup\n[rank3]: self._model = self._setup_model(\n[rank3]: ^^^^^^^^^^^^^^^^^^\n[rank3]: File \"/home/allen.philip/src/finetune/torchtune/.venv/lib/python3.11/site-packages/recipes/lora_finetune_distributed.py\", line 525, in _setup_model\n[rank3]: base_missing, base_unexpected = training.load_from_full_model_state_dict(\n[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n[rank3]: File \"/home/allen.philip/src/finetune/torchtune/.venv/lib/python3.11/site-packages/torchtune/training/_distributed.py\", line 455, in load_from_full_model_state_dict\n[rank3]: return model.load_state_dict(sharded_sd, strict=strict, assign=True)\n[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n[rank3]: File \"/home/allen.philip/src/finetune/torchtune/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py\", line 2593, in load_state_dict\n[rank3]: raise RuntimeError(\n[rank3]: RuntimeError: Error(s) in loading state_dict for FSDPTransformerDecoder:\n[rank3]: size mismatch for layers.0._checkpoint_wrapped_module.attn.output_proj.weight: copying a param with shape torch.Size([5", "environment": { "gpu_type": "L40" }, "fix": null, "verified_fix": false, "framework": "FSDP", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_2830", "source_url": "https://github.com/meta-pytorch/torchtune/issues/2830", "source_type": "github_issue", "failure_type": "OOM_FRAGMENTATION", "title": "OOM handling and recovery", "error_message": "We just hit OOM, revealing that by default torchtune does not use torch.compile and that it does not use fused linear cross entropy yet...\n\nI found the following report from 2024:\n- https://www.reddit.com/r/LocalLLaMA/comments/1di0fhv/torchtune_vs_axolotl_vs_unsloth_trainer/\n- https://wandb.ai/augmxnt/train-bench/reports/Trainer-performance-comparison-torchtune-vs-axolotl-vs-Unsloth---Vmlldzo4MzU3NTAx\n\nAre there any plans to make torchtune excellent for peak GPU memory usage and practical OOM ha", "environment": { "cuda_version": "5.15" }, "fix": null, "verified_fix": false, "framework": "Axolotl", "confidence": 0.6, "is_cascade": false, "raw_body_length": 4504 }, { "failure_id": "lf_github_issue_2658", "source_url": "https://github.com/meta-pytorch/torchtune/issues/2658", "source_type": "github_issue", "failure_type": "CHECKPOINT_CORRUPTION", "title": "How to QAT the Llama-3b backbone model?", "error_message": "Traceback (most recent call last):\n```", "environment": {}, "fix": null, "verified_fix": false, "framework": "FSDP", "confidence": 0.6, "is_cascade": true, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_45923", "source_url": "https://github.com/huggingface/transformers/issues/45923", "source_type": "github_issue", "failure_type": "GRADIENT_EXPLOSION", "title": "Nemotron-3-Nano-Omni: supports_gradient_checkpointing flag missing on trust_remote_code variant (1-line fix)", "error_message": "ValueError`, even though the block-level machinery (`NemotronHBlock(GradientCheckpointingLayer)`) is already in place.\n\n**Minimal repro:**\n\n```python\nfrom transformers import AutoModelForCausalLM\nimport torch\n\nmodel = AutoModelForCausalLM.from_pretrained(\n \"nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16\",\n trust_remote_code=True,\n torch_dtype=torch.bfloat16,\n device_map=\"cuda:0\",\n)\n\nmodel.gradient_checkpointing_enable()\n# → ValueError: NemotronHForCausalLM does not support gradie", "environment": { "pytorch_version": "2.10.0", "cuda_version": "12.8" }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_45663", "source_url": "https://github.com/huggingface/transformers/issues/45663", "source_type": "github_issue", "failure_type": "FSDP_ERROR", "title": "Gemma-4 training with FSDP2 raises `KeyError` in `Gemma4TextAttention.forward` because `shared_kv_states` is rebuilt per-layer", "error_message": "nccl\")\ntorch.cuda.set_device(dist.get_rank())\ncast = os.environ.get(\"CAST_FORWARD_INPUTS\", \"1\") == \"1\"\nprint(f\"cast_forward_inputs={cast}\", flush=True)\n\n# 2-layer Gemma-4: layer 0 writes shared_kv_states[0], layer 1 reads it.\n# * num_kv_shared_layers=1 — default 0 means no sharing layer, bug can't fire.\n# * layer_types must be set (default None breaks model init); both must be\n# full_attention because Gemma-4 force-overrides the last layer to full,\n# so the first must match for the s", "environment": { "pytorch_version": "2.10.0", "gpu_type": "H100" }, "fix": null, "verified_fix": false, "framework": "DeepSpeed", "confidence": 0.6, "is_cascade": true, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_45923", "source_url": "https://github.com/huggingface/transformers/issues/45923", "source_type": "github_issue", "failure_type": "GRADIENT_EXPLOSION", "title": "Nemotron-3-Nano-Omni: supports_gradient_checkpointing flag missing on trust_remote_code variant (1-line fix)", "error_message": "ValueError`, even though the block-level machinery (`NemotronHBlock(GradientCheckpointingLayer)`) is already in place.\n\n**Minimal repro:**\n\n```python\nfrom transformers import AutoModelForCausalLM\nimport torch\n\nmodel = AutoModelForCausalLM.from_pretrained(\n \"nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16\",\n trust_remote_code=True,\n torch_dtype=torch.bfloat16,\n device_map=\"cuda:0\",\n)\n\nmodel.gradient_checkpointing_enable()\n# → ValueError: NemotronHForCausalLM does not support gradie", "environment": { "pytorch_version": "2.10.0", "cuda_version": "12.8" }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_45663", "source_url": "https://github.com/huggingface/transformers/issues/45663", "source_type": "github_issue", "failure_type": "FSDP_ERROR", "title": "Gemma-4 training with FSDP2 raises `KeyError` in `Gemma4TextAttention.forward` because `shared_kv_states` is rebuilt per-layer", "error_message": "nccl\")\ntorch.cuda.set_device(dist.get_rank())\ncast = os.environ.get(\"CAST_FORWARD_INPUTS\", \"1\") == \"1\"\nprint(f\"cast_forward_inputs={cast}\", flush=True)\n\n# 2-layer Gemma-4: layer 0 writes shared_kv_states[0], layer 1 reads it.\n# * num_kv_shared_layers=1 — default 0 means no sharing layer, bug can't fire.\n# * layer_types must be set (default None breaks model init); both must be\n# full_attention because Gemma-4 force-overrides the last layer to full,\n# so the first must match for the s", "environment": { "pytorch_version": "2.10.0", "gpu_type": "H100" }, "fix": null, "verified_fix": false, "framework": "DeepSpeed", "confidence": 0.6, "is_cascade": true, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_45161", "source_url": "https://github.com/huggingface/transformers/issues/45161", "source_type": "github_issue", "failure_type": "DEVICE_ASSERT", "title": "Only TP not working with GPT-OSS MoE model", "error_message": "Traceback (most recent call last):\n[rank0]: File \"/local/mnt/workspace/OG_TP/train_tp_trainer.py\", line 104, in \n[rank0]: trainer.train()\n[rank0]: File \"/local/mnt/workspace/OG_TP/tr", "environment": {}, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": true, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_44928", "source_url": "https://github.com/huggingface/transformers/issues/44928", "source_type": "github_issue", "failure_type": "GRADIENT_EXPLOSION", "title": "[Bug] Catastrophic gradient explosion (NaN) in RLHF with Qwen3.5 due to 3D position_ids forcing SDPA Math fallback and BF16 collapse", "error_message": "### System Info\n\nCopy-and-paste the text below in your GitHub issue and FILL OUT the two last points.\n\n- `transformers` version: 5.3.0\n- Platform: Linux-6.8.0-40-generic-x86_64-with-glibc2.35\n- Python version: 3.11.15\n- Huggingface_hub version: 1.7.1\n- Safetensors version: 0.7.0\n- Accelerate version: 1.13.0\n- Accelerate config: not found\n- DeepSpeed version: 0.18.8\n- PyTorch version (accelerator?): 2.10.0+cu128 (CUDA)\n- Using distributed or parallel set-up in script?: \n- Using GPU in", "environment": { "pytorch_version": "2.10.0", "cuda_version": "12.8", "gpu_type": "A100" }, "fix": null, "verified_fix": false, "framework": "DeepSpeed", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_44368", "source_url": "https://github.com/huggingface/transformers/issues/44368", "source_type": "github_issue", "failure_type": "OOM_FRAGMENTATION", "title": "when using ms-swift lora fine-tuning Qwen3.5-27B, each layer emits warning:You should update the config with `tie_word_embeddings=False` to silence this warning", "error_message": "### System Info\n\ntransformers==5.2.0\ntorch==2.8.0\ndeepspeed==0.18.6\npython==3.10\nms-swift==4.0.0.dev0\n\n### Who can help?\n\n_No response_\n\n### Information\n\n- [x] The official example scripts\n- [x] My own modified scripts\n\n### Tasks\n\n- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)\n- [ ] My own task or dataset (give details below)\n\n### Reproduction\n\n# 4 * 30GiB\nPYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \\\nNPROC_PER_NODE=2 \\\nMAX_PIXELS=1003520 \\\nVIDEO_MAX", "environment": {}, "fix": null, "verified_fix": false, "framework": "DeepSpeed", "confidence": 0.6, "is_cascade": false, "raw_body_length": 1674 }, { "failure_id": "lf_github_issue_43856", "source_url": "https://github.com/huggingface/transformers/issues/43856", "source_type": "github_issue", "failure_type": "OOM_FRAGMENTATION", "title": "Inefficient memory usage during Qwen3 MoE training", "error_message": "### System Info\n\n- `transformers` version: 4.57.3\n- Platform: Linux-6.8.0-90-generic-x86_64-with-glibc2.35\n- Python version: 3.12.12\n- Huggingface_hub version: 0.36.0\n- Safetensors version: 0.7.0\n- Accelerate version: 1.12.0\n- Accelerate config: not found\n- DeepSpeed version: not installed\n- PyTorch version (accelerator?): 2.9.0+cu126 (CUDA)\n- Tensorflow version (GPU?): not installed (NA)\n- Flax version (CPU?/GPU?/TPU?): not installed (NA)\n- Jax version: not installed\n- JaxLib version: not in", "environment": { "pytorch_version": "2.9.0", "cuda_version": "0.8", "gpu_type": "H100" }, "fix": null, "verified_fix": false, "framework": "DeepSpeed", "confidence": 0.6, "is_cascade": false, "raw_body_length": 4055 }, { "failure_id": "lf_github_issue_43541", "source_url": "https://github.com/huggingface/transformers/issues/43541", "source_type": "github_issue", "failure_type": "TORCH_COMPILE_DYNAMO", "title": "RuntimeError: MixtralForCausalLM float32 model errors at grouped_mm op during torch dynamo tracing", "error_message": "RuntimeError, cond, message)\n File \"/home/ubuntu/workplace/tc_moduscope/src/TorchNeuronEager/.venv/lib/python3.10/site-packages/torch/__init__.py\", line 1677, in _check_with\n raise error_type(message_evaluated)\ntorch._dynamo.exc.TorchRuntimeError: Dynamo failed to run FX node with fake tensors: call_function (*(FakeTensor(..., size=(256, 256)), FakeTensor(..., size=(8, 256, 512), requires_grad=True)), **{'offs': FakeTensor(..., si", "environment": { "pytorch_version": "2.9.1" }, "fix": null, "verified_fix": false, "framework": "DeepSpeed", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_3265", "source_url": "https://github.com/huggingface/peft/issues/3265", "source_type": "github_issue", "failure_type": "GRADIENT_EXPLOSION", "title": "prepare_model_for_kbit_training adds ~1 GB CUDA reserved memory in 500 ms — undocumented cost that breaks memory-constrained training on 8 GB unified-memory devices", "error_message": "torch.cuda.memory_allocated() // (1024 * 1024),\n \"cuda_reserved_mb\": torch.cuda.memory_reserved() // (1024 * 1024),\n }), flush=True)\n\nbnb_cfg = BitsAndBytesConfig(\n load_in_4bit=True,\n bnb_4bit_quant_type=\"nf4\",\n bnb_4bit_compute_dtype=torch.bfloat16,\n bnb_4bit_use_double_quant=True,\n)\n\nsnap(\"00_start\")\nmodel = AutoModelForCausalLM.from_pretrained(\n \"mistralai/Mistral-7B-v0.3\",\n quantization_config=bnb_cfg,\n torch_dtype=torch.bfloat16,\n device_map={\"\": 0},\n ", "environment": { "cuda_version": "0.3", "gpu_type": "RTX 4060" }, "fix": null, "verified_fix": false, "framework": "FSDP", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_3169", "source_url": "https://github.com/huggingface/peft/issues/3169", "source_type": "github_issue", "failure_type": "DDP_ERROR", "title": "LoRA + BnB INT8 + CPU offload: output tensor on wrong device in tuners/lora/bnb.py", "error_message": "RuntimeError: Expected all tensors to be on the same device,\nbut found at least two devices, cuda:0 and cpu!\n```\n\n## Full context\n\nDiscovered while making Gemma4 26B-A4B train on a single RTX 4090 (BnB INT8 + LoRA + Gradient Checkpointing + CPU offload). All patches + complete training example:\n\nhttps://github.com/sirfyyn/consumer-llm-patches\n\nHappy to submit a PR. The fix is a one-liner per site but needs testing against non-offload setups to confirm no regression.", "environment": { "gpu_type": "RTX 4090" }, "fix": null, "verified_fix": false, "framework": "Unknown", "confidence": 0.6, "is_cascade": false, "raw_body_length": 1345 }, { "failure_id": "lf_github_issue_3073", "source_url": "https://github.com/huggingface/peft/issues/3073", "source_type": "github_issue", "failure_type": "GRADIENT_EXPLOSION", "title": "LoRA gradients not normalized by input norm → training instability (NaN)", "error_message": "### System Info\n\nFound an issue: PEFT scales LoRA output by α / r​, but gradients remain linearly dependent on input norm ∥x∥∥x∥:\n∥Grad∥∝α / r⋅∥x∥ \nWhen activations vary 10–100× across layers, this causes:\n Gradient explosion → NaN on first steps\n Vanishing gradients in layers with small activations\nWorkaround: Added a hook that normalizes gradients by input energy:\nscale=α / r * mean(x^T*x)​\nWithout this — NaN on step 1–5. With this — stable training.\nSuggestion: Add normalize_grad_by_inp", "environment": {}, "fix": null, "verified_fix": false, "framework": "Unknown", "confidence": 0.6, "is_cascade": false, "raw_body_length": 3530 }, { "failure_id": "lf_github_issue_13696", "source_url": "https://github.com/huggingface/diffusers/issues/13696", "source_type": "github_issue", "failure_type": "DDP_ERROR", "title": "[bug] The mask is not correctly sharded for QwenImageTransformer + Ulysses SP", "error_message": "Traceback (most recent call last):\n[rank0]: File \"/scratch/fq9hpsac/mikecheung/gitlocal/verl-omni/test.py\", line 96, in \n[rank0]: main()\n[rank0]: File \"/scratch/fq9hpsac/mikecheung/gitlocal/verl-omni/test.py\", line 89, in main\n[rank0]: torch.testing.assert_close(output_sp.float(), output_no_sp.float(), rtol=1e-2, atol=1e-2)\n[rank0]: File \"/scratch/fq9hpsac/mikecheung/miniforge3/envs/verl-omni/lib/python3.12/site-packages/torch/testing/_comparison.py\", line 1600, in assert_close\n[rank0]: raise error_metas[0].to_error(msg)\n[rank0]: AssertionError: Tensor-likes are not close!", "environment": { "num_gpus": 2 }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": true, "raw_body_length": 4588 }, { "failure_id": "lf_github_issue_13624", "source_url": "https://github.com/huggingface/diffusers/issues/13624", "source_type": "github_issue", "failure_type": "DEVICE_ASSERT", "title": "aura_flow model/pipeline review", "error_message": "ValueError(\n f\"Input patch grid ({h_p}, {w_p}) exceeds AuraFlow positional embedding grid ({h_max}, {w_max}).\"\n )\n```\n\n## Issue 2: `out_channels=None` is serialized but crashes in `forward`\n\nAffected code:\nhttps://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/auraflow_transformer_2d.py#L319-L320\nhttps://github.com/huggingface/diffusers/blob/0f1abc4ae8b0eb2a3b40e82a310507281144c423/src/diffusers/models/transformers/auraf", "environment": {}, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_7961", "source_url": "https://github.com/deepspeedai/DeepSpeed/issues/7961", "source_type": "github_issue", "failure_type": "VERSION_MISMATCH", "title": "[BUG][ZeRO-3 / BF16] No public API to reassemble optimizer moments (exp_avg, exp_avg_sq) from sharded checkpoint files post-hoc", "error_message": "**Describe the bug**\nAfter training with ZeRO Stage 3 + BF16, we need to extract Adam optimizer moments (exp_avg, exp_avg_sq) from saved checkpoint files for downstream analysis (e.g. loss landscape studies). There is no documented public API to do this safely. \noptimizer_state_dict\n └── optimizer_state_dict\n └── state\n └── 0 ← single entry for ALL params\n exp_avg Tensor[33_554_432] float32\n exp_avg_sq Tensor[33_5", "environment": { "cuda_version": "11.8", "gpu_type": "A100" }, "fix": null, "verified_fix": false, "framework": "DeepSpeed", "confidence": 0.6, "is_cascade": false, "raw_body_length": 4076 }, { "failure_id": "lf_github_issue_7844", "source_url": "https://github.com/deepspeedai/DeepSpeed/issues/7844", "source_type": "github_issue", "failure_type": "SILENT_HANG", "title": "[BUG] DeepSpeed ZeRO-3 deadlock in engine.step() at step 0 under 2-GPU execution (RTX 3090, torch 2.2.1, DS 0.14.2)", "error_message": "NCCL_SOCKET_IFNAME=eno2\nexport GLOO_SOCKET_IFNAME=eno2\nexport NCCL_IB_DISABLE=1\n\n# ZeRO stage\nexport DS_ZERO_STAGE=3\n\n# run\ndeepspeed --num_gpus 2 testcases/deepspeed_testcase.py\n```\n\n3. Observe logs: \nBoth ranks print:\n```\n- R0 STEP0 route=0 -> forward\n- R1 STEP0 route=0 -> forward\n- ... -> backward\n- R0 STEP0 route=0 -> step_begin\n- R1 STEP0 route=0 -> step_begin\nThen repeated:\n- HEARTBEAT rank=0 step=0 phase=step_begin age=...\n- HEARTBEAT rank=1 step=0 phase=step_begin age=...\nAfter >60s:\n- T", "environment": { "pytorch_version": "2.2.1", "cuda_version": "12.1", "gpu_type": "RTX 3090", "num_gpus": 2 }, "fix": null, "verified_fix": false, "framework": "DeepSpeed", "confidence": 0.6, "is_cascade": true, "raw_body_length": 3989 }, { "failure_id": "lf_github_issue_7811", "source_url": "https://github.com/deepspeedai/DeepSpeed/issues/7811", "source_type": "github_issue", "failure_type": "DEEPSPEED_ERROR", "title": "[BUG] ZeRO-3: zero.GatheredParameters([multiple params], modifier_rank=None) + in-place slice touch triggers assert not param.ds_active_sub_modules in free_param()", "error_message": "nccl\")\n\n rank = dist.get_rank()\n world = dist.get_world_size()\n\n local_rank = int(os.environ.get(\"LOCAL_RANK\", \"0\"))\n torch.cuda.set_device(local_rank)\n device = torch.device(f\"cuda:{local_rank}\")\n\n def barrier():\n dist.barrier()\n\n ds_config = os.environ.get(\"DEEPSPEED_CONFIG\", \"ds_config_zero3_stress.json\")\n\n VOCAB = env_int(\"VOCAB\", 32768)\n D_MODEL = env_int(\"D_MODEL\", 2048)\n ITERS = env_int(\"ITERS\", 200)\n TILE = env_int(\"TILE\", 8)\n DO_BWD = env_bool(", "environment": {}, "fix": null, "verified_fix": false, "framework": "DeepSpeed", "confidence": 0.6, "is_cascade": true, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_7746", "source_url": "https://github.com/deepspeedai/DeepSpeed/issues/7746", "source_type": "github_issue", "failure_type": "TORCH_COMPILE_DYNAMO", "title": "[BUG] Dtype mismatch (bf16 vs fp32) when resuming Muon optimizer from checkpoint", "error_message": "RuntimeError` occurs due to a dtype mismatch.\n- Model parameters and gradients are in `bf16`.\n- Optimizer state (`momentum_buffer`) is loaded from the checkpoint as `fp32`.\n- The mismatch happens when Muon tries to apply updates (e.g., `lerp_`) between `fp32` momentum buffers and `bf16` gradients.\n\n## Minimal Reproducible Example\n\n```python\nimport torch\nimport os\nimport deepspeed\n\ntorch.cuda.set_device(int(os.environ[\"LOCAL_RANK\"]))\n\ndef train_step(model_engine, x, y):\n output = model_engine(", "environment": { "cuda_version": "0.050000000000000044", "gpu_type": "H100" }, "fix": null, "verified_fix": false, "framework": "DeepSpeed", "confidence": 0.6, "is_cascade": false, "raw_body_length": 3073 }, { "failure_id": "lf_github_issue_7741", "source_url": "https://github.com/deepspeedai/DeepSpeed/issues/7741", "source_type": "github_issue", "failure_type": "CHECKPOINT_CORRUPTION", "title": "[BUG] DecoupledCheckpointEngine hangs indefinitely due to missing timeouts and process health checks", "error_message": "ValueError(f\"Checkpoint commit info mismatch: expected {self.commit_info}, got {info}\")\n\n# Add timeout to event wait with process health check\nTIMEOUT_SECONDS = 300 # 5 minutes\nwhile not self.save_event.wait(timeout=10):\n if not self.ckpt_process.is_alive():\n raise RuntimeError(\"Checkpoint process died unexpectedly\")\n\n# Add timeout to process join\nself.ckpt_process.join(timeout=TIMEOUT_SECONDS)\nif self.ckpt_process.is_alive():\n self.ckpt_process.terminate()\n raise RuntimeError(\"", "environment": {}, "fix": null, "verified_fix": false, "framework": "DeepSpeed", "confidence": 0.6, "is_cascade": false, "raw_body_length": 2963 }, { "failure_id": "lf_github_issue_7697", "source_url": "https://github.com/deepspeedai/DeepSpeed/issues/7697", "source_type": "github_issue", "failure_type": "OOM_FRAGMENTATION", "title": "[BUG] Significant Performance Difference between DeepSpeed's zero_stage=1 and zero_stage=2", "error_message": "**Describe the bug**\nI am using the `SFTTrainer` of Huggingface's [TRL](https://github.com/huggingface/trl), and I found the training and evaluation exhibit significant difference between `zero_stage=1` and `zero_stage=2`, shown as below:\n\n\"Image\"\n\n\"Image\"\n[rank0]: main()\n[rank0]: File \"/inspire/ssd/project/robot3d/mazipei-253107140027/WorldActionModel/main.py\", line 45, in main\n[rank0]: runner.train()\n[rank0]: File \"/inspire/ssd/project/robot3d/mazipei-253107140027/WorldActionModel/runner/ltx_video_trainer.py\", line 489, in train\n[rank0]: self.state.accelerator.load_state(resume_dir, weights_only=False)\n[rank0]: File \"/opt/miniconda3/envs/genie_envisioner/lib/python3.10/site-packages/accelerate/accelerator.py\", line 3690, in load_state\n[rank0]: model.load_checkpoint(input_dir, ckpt_id, **load_model_func_kwargs)\n[rank0]: TypeError: DeepSpeedEngine.load_checkpoint() got an unexpected keyword argument 'weights_only'\n```", "environment": { "gpu_type": "A100" }, "fix": null, "verified_fix": false, "framework": "DeepSpeed", "confidence": 0.6, "is_cascade": false, "raw_body_length": 2599 }, { "failure_id": "lf_github_issue_7607", "source_url": "https://github.com/deepspeedai/DeepSpeed/issues/7607", "source_type": "github_issue", "failure_type": "VERSION_MISMATCH", "title": "[BUG] ZeRO 2 ipg_buckets key error", "error_message": "NCCL version: the result is overridden to be `fp32` in any case.\n\nNote: It appears with torch autocast enabled, a different initialisation path might avoid this crash.\n", "environment": { "cuda_version": "12.9" }, "fix": null, "verified_fix": false, "framework": "DeepSpeed", "confidence": 0.6, "is_cascade": true, "raw_body_length": 4672 }, { "failure_id": "lf_github_issue_7584", "source_url": "https://github.com/deepspeedai/DeepSpeed/issues/7584", "source_type": "github_issue", "failure_type": "CHECKPOINT_CORRUPTION", "title": "[BUG] universal checkpoin for stage3 fail when there are multiple subgroups", "error_message": "Traceback (most recent call last):\n File \"/work/zhengchenyu/ai-project/qwen3/scripts/ds_to_universal_example.py\", line 21, in \n main()\n File \"/work/zhengchenyu/ai-project/qwen3/scripts/ds_to_universal_example.py\", line 18, in main\n ds_to_universal_main(args)\n File \"/opt/conda/lib/python3.11/site-packages/deepspeed/checkpoint/ds_to_universal.py\", line 523, in main\n _extract_zero_shard_files_stage3(args, optim_files, param_shapes, dp_degree, temp_dir)\n File \"/opt/conda/lib/python3.11/site-packages/deepspeed/checkpoint/ds_to_universal.py\", line 375, in _extract_zero_shard_files_stage3\n _do_parallel_work(do_work, list(range(dp_degree)), args.num_extract_workers)\n File \"/opt/conda/lib/python3.11/site-packages/deepspeed/checkpoint/ds_to_universal.py\", line 359, in _do_parallel_work\n results.append(do_work(work))\n ^^^^^^^^^^^^^\n File \"/opt/conda/lib/python3.11/site-packages/deepspeed/checkpoint/ds_to_universal.py\", line 167, in extract_zero_shards_stage3\n dump_param_fragment(temp_dir, 0, dp_index, state_key, flat_state[state_key], name, offset,\n File \"/opt/conda/lib/python3.11/site-packages/deepspeed/checkpoint/ds_to_universal.py\", line 194, in dump_param_fragment\n state_flat_tensor = state_flat_tensor.narrow(0, offset, numel).clone()\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nRuntimeError: start (0) + length (155582464) exceeds dimension size (74499072).", "environment": {}, "fix": null, "verified_fix": false, "framework": "DeepSpeed", "confidence": 0.6, "is_cascade": false, "raw_body_length": 2364 }, { "failure_id": "lf_github_issue_7581", "source_url": "https://github.com/deepspeedai/DeepSpeed/issues/7581", "source_type": "github_issue", "failure_type": "VERSION_MISMATCH", "title": "[BUG] Unit Test TestFp8ComposabilityAcrossZero.test[fp16] failure", "error_message": "Traceback (most recent call last):\n File \"/usr/lib/python3.12/multiprocessing/pool.py\", line 125, in worker\n result = (True, func(*args, **kwds))\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.12/multiprocessing/pool.py\", line 51, in starmapstar\n return list(itertools.starmap(args[0], args[1]))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/root/not_working/DeepSpeed/tests/unit/common.py\", line 325, in _dist_run\n raise e\n File \"/root/not_working/DeepSpeed/tests/unit/common.py\", line 317, in _dist_run\n self.run(**self._fixture_kwargs)\n File \"/root/not_working/DeepSpeed/tests/unit/common.py\", line 473, in run\n self._current_test(**fixture_kwargs)\n File \"/root/not_working/DeepSpeed/tests/unit/runtime/half_precision/test_fp8.py\", line 98, in test\n assert (all_equal)\n ^^^^^^^^^\nAssertionError\n\"\"\"\n```", "environment": { "cuda_version": "13.0" }, "fix": null, "verified_fix": false, "framework": "DeepSpeed", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_7549", "source_url": "https://github.com/deepspeedai/DeepSpeed/issues/7549", "source_type": "github_issue", "failure_type": "VERSION_MISMATCH", "title": "[BUG] save_checkpoint race when consolidating NVMe offloaded tensors → FileExistsError", "error_message": "**Describe the bug**\nWhen calling save_checkpoint (via accelerator.save_state) from all ranks on a multi-GPU run with ZeRO Stage 3 NVMe offloading enabled, DeepSpeed attempts to consolidate per-rank NVMe offload directories into a single shared offloaded_tensors destination using shutil.copytree. Because all ranks call the method concurrently, two processes race to create/copy into the same destination directory and one fails with FileExistsError: [Errno 17] File exists. The docstring correctly ", "environment": { "cuda_version": "12.4", "num_gpus": 2 }, "fix": null, "verified_fix": false, "framework": "DeepSpeed", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_7546", "source_url": "https://github.com/deepspeedai/DeepSpeed/issues/7546", "source_type": "github_issue", "failure_type": "VERSION_MISMATCH", "title": "[BUG] Zero 3 checkpoints not saving model state when load_universal=True", "error_message": "**Describe the bug**\nWhen doing model_engine.save_checkpoint, when zero is set to stage 3, and \"load_universal=True\" the model state files do not appear to get saved (except on rank 0).\n\nAll `bf16_zero_pp_rank_X_mp_rank_00_optim_states.pt` files seem correctly saved. But only a single `zero_pp_rank_0_mp_rank_00_model_states.pt` is saved, and none for other ranks. Both the `_to_universal` or `_to_fp32` checkpoint scripts fail due to missing model state.\n\nCommenting out the\n```\n[deepspeed.checkpoi", "environment": { "cuda_version": "12.8" }, "fix": null, "verified_fix": false, "framework": "DeepSpeed", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_7533", "source_url": "https://github.com/deepspeedai/DeepSpeed/issues/7533", "source_type": "github_issue", "failure_type": "SILENT_HANG", "title": "[BUG]save model NCCL timeout", "error_message": "nccl_timeout more longer?\n\n**To Reproduce**\nSteps to reproduce the behavior:\n1. Go to '...'\n2. Click on '....'\n3. Scroll down to '....'\n4. See error\n\n**Expected behavior**\nwhen init：\n deepspeed.init_distributed(timeout=timedelta(minutes=60))\nwhen save model：\n model_to_save.save_pretrained(output_dir, state_dict=output_state_dict, **kwargs)\n\n\n\n**ds_report output**\nPlease run `ds_report` to give us details about your setup.\n\n**Screenshots**\n[rank5]:[E901 05:20:03.110823792 ProcessGr", "environment": {}, "fix": null, "verified_fix": false, "framework": "DeepSpeed", "confidence": 0.6, "is_cascade": true, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_7482", "source_url": "https://github.com/deepspeedai/DeepSpeed/issues/7482", "source_type": "github_issue", "failure_type": "OOM_FRAGMENTATION", "title": "[BUG] GPU OOM when finetune Qwen2.5-14B with ZeRO2+offload on 4xA100 40G cards", "error_message": "**Describe the bug**\nWhen finetune Qwen2.5-14B with ZeRO2+offload on 4xA100 40G cards, got GPU OOM error.\n\n**To Reproduce**\nConfig file:\n```\n{\n \"train_batch_size\": 8,\n \"bf16\": { \"enabled\": true },\n \"zero_optimization\": {\n \"stage\": 2,\n \"offload_optimizer\": {\n \"device\": \"cpu\",\n \"pin_memory\": false\n }\n },\n \"optimizer\": {\n \"type\": \"AdamW\",\n \"params\": {\n \"lr\": 2e-5,\n \"betas\": [0.9, 0.999],\n \"eps\": 1e-8,\n \"weight_decay\":", "environment": { "cuda_version": "20.00", "gpu_type": "A100" }, "fix": null, "verified_fix": false, "framework": "DeepSpeed", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_3453", "source_url": "https://github.com/axolotl-ai-cloud/axolotl/issues/3453", "source_type": "github_issue", "failure_type": "OOM_FRAGMENTATION", "title": "Sample Packing cause loss 0 and ppl 1 for Qwen35", "error_message": "NCCL_ASYNC_ERROR_HANDLING=1\nexport TORCH_NCCL_BLOCKING_WAIT=1\nexport CUDA_HOME=/usr/local/cuda-12.4\nexport TORCH_CUDA_ARCH_LIST=\"8.0;8.6;9.0\"\nexport AXOLOTL_DO_NOT_TRACK=1\n\nDATE=$(date +%Y%m%d_%H%M)\nLOG_DIR=logs\n\nLOG_FILE=$LOG_DIR/qwen35_9B_10M_$DATE.log\n#############################\n# Train\n#############################\n\nCUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 accelerate launch \\\n --main_process_port 29502 \\\n --num_processes 8 \\\n --num_machines 1 \\\n --mixed_precision bf16 \\\n --dynam", "environment": { "cuda_version": "12.4" }, "fix": null, "verified_fix": false, "framework": "DeepSpeed", "confidence": 0.6, "is_cascade": true, "raw_body_length": 4478 }, { "failure_id": "lf_github_issue_21611", "source_url": "https://github.com/Lightning-AI/pytorch-lightning/issues/21611", "source_type": "github_issue", "failure_type": "OOM_FRAGMENTATION", "title": "cache_enabled=False in autocast causes OOM regression for iterative decoding workloads", "error_message": "### Bug description\n\n Lightning 2.6.0 introduced cache_enabled=False in MixedPrecision.autocast_context_manager (compared to 2.5.x which used the default \n cache_enabled=True):\n \n # 2.5.x \n``` \n def autoc", "environment": { "cuda_version": "158.00" }, "fix": null, "verified_fix": false, "framework": "PyTorch Lightning", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_21431", "source_url": "https://github.com/Lightning-AI/pytorch-lightning/issues/21431", "source_type": "github_issue", "failure_type": "CHECKPOINT_CORRUPTION", "title": "Trainer.save_checkpoint() occasionally saves corrupt checkpoint in Docker/WSL", "error_message": "### Bug description\n\n\nAn exception may occur when loading a model checkpoint (with `LightningModule.load_from_checkpoint()`) that was corrupted during saving with PyTorch Lightning (`Trainer.save_checkpoint()`).\n\nTrying to load the checkpoint file directly e.g. with `torch.load()` shows the same exception. Comparing the file with a valid checkpoint (e.g. with `diff`) confirms that both differ. This indicates that the checkpoint file is corrupted during saving.\n\nThis exception appears for CPU and", "environment": { "pytorch_version": "2.9.0", "cuda_version": "2.9" }, "fix": null, "verified_fix": false, "framework": "PyTorch Lightning", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_21406", "source_url": "https://github.com/Lightning-AI/pytorch-lightning/issues/21406", "source_type": "github_issue", "failure_type": "CHECKPOINT_CORRUPTION", "title": "Checkpointing in signal handlers for SLURM auto-requeueing leads to intermittent failures", "error_message": "### Bug description\n\nI have been experienced intermittent and difficult to pin down failures when lightning tried to auto-requeue my jobs on our SLURM cluster on time out. After some debugging (`print` in the signal handler for `SIGUSR1` and many runs), I saw that sometimes the handler would just stop running after or while saving the HPC checkpoint to disk.\n\nSignal handlers are a pretty special environment, because they can run after any python bytecode instruction, so also in the middle of oth", "environment": { "pytorch_version": "2.6.0", "cuda_version": "12.6" }, "fix": null, "verified_fix": false, "framework": "PyTorch Lightning", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_2158", "source_url": "https://github.com/NVIDIA/nccl/issues/2158", "source_type": "github_issue", "failure_type": "STRAGGLER", "title": "[Question]: Two nodes, 8 GPUs + 8 RoCE NICs per node, all on one switch: same VLAN and subnet okay?", "error_message": "NCCL collectives (e.g., correctness of communication), or other problems such as increased latency, PFC deadlocks, congestion, or performance degradation?\n\nI understand that RoCE typically benefits from careful traffic isolation to avoid head‑of‑line blocking. However, given that:\n\n- All NICs are under the same physical switch,\n\n- There are only two nodes, and\n\n- Communication patterns can be symmetric (e.g., all‑to‑all),\n\nis there any NCCL‑specific reason to split them into multiple VLANs/subne", "environment": { "num_gpus": 8 }, "fix": null, "verified_fix": false, "framework": "Unknown", "confidence": 0.6, "is_cascade": true, "raw_body_length": 1129 }, { "failure_id": "lf_github_issue_186357", "source_url": "https://github.com/pytorch/pytorch/issues/186357", "source_type": "github_issue", "failure_type": "TORCH_COMPILE_AOTI", "title": "AOTInductor on Windows: models with >2GB constants fail to load — 32-bit `lseek` overflow (`weights_offset must be aligned to 16K boundary`)", "error_message": "RuntimeError: weights_offset must be aligned to 16K boundary` (or, depending on the\nlow-bit luck of the corrupted offset, loads and then hits an illegal memory access at run). The\nweights are in fact laid out at a correctly 16K-aligned file offset — the runtime just **mis-reads\nthe file size** because `model_base.h` uses 32-bit `lseek`, which overflows for files larger than\n2 GB on Windows/MSVC.\n\nThis is distinct from #145610 (ARM 64K-page granularity); same assertion message, different cause.\n\n", "environment": { "cuda_version": "12.8", "gpu_type": "RTX 5060" }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 3644 }, { "failure_id": "lf_github_issue_186216", "source_url": "https://github.com/pytorch/pytorch/issues/186216", "source_type": "github_issue", "failure_type": "TORCH_COMPILE_AOTI", "title": "Bug ，Cannot compile Native C++ API to export LibTorch trained models directly to AOTInductor format", "error_message": "### 🐛 Describe the bug\n\n```\n\n### Versions\n\nWe are requesting a supported pathway or native C++ API to export models trained purely in C++ (LibTorch) directly to the AOTInductor (.pt2) format, allowing them to be loaded via torch::inductor::AOTIModelPackageLoader without requiring a Python-based graph recreation step.\n\nMotivation\nIn our engineering stack, we have invested heavily in pure C++ LibTorch pipelines for both training and inference. We manage complex native memory environments, strict m", "environment": {}, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 3023 }, { "failure_id": "lf_github_issue_186213", "source_url": "https://github.com/pytorch/pytorch/issues/186213", "source_type": "github_issue", "failure_type": "OOM_FRAGMENTATION", "title": "CUDA expandable IPC receiver reserves ~9/8 of total GPU memory VA per imported handle and can exhaust device virtual address space", "error_message": "torch.cuda.memory_allocated()` and `torch.cuda.memory_reserved()` stay at `0` in the receiver, and `mem_get_info()` still reports free memory.\n\nObserved failure:\n\n```text\nRuntimeError: CUDA driver error: invalid argument\n```\n\nWith local instrumentation around `getIpcDevPtr` and `ExpandableSegment::fromShared`, I observed the failure at:\n\n```text\n[CUDA IPC cache] miss-begin hits=0 misses=2621 inserts=2620 erases=0 failures=0 cache_size=2620 handle_size=30 device=0\n[CUDA IPC expandable] fromShared", "environment": {}, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_185839", "source_url": "https://github.com/pytorch/pytorch/issues/185839", "source_type": "github_issue", "failure_type": "OOM_FRAGMENTATION", "title": "torch.linalg.qr (float64) repeated in a loop crashes the process (SIGSEGV, exit 139) on ROCm 7.x", "error_message": "torch.cuda.memory_reserved()` at the crash point is **0.02 GB** (out of 94 GB). The matrices are tiny and freed each iteration.\n\n## Workarounds (both confirmed, 60/60 iterations pass)\n\n```python\n# (1) drain pending events periodically\nfor i in range(50):\n a = torch.randn(512, 512, device='cuda', dtype=torch.float64)\n torch.linalg.qr(a)\n if (i + 1) % 20 == 0:\n torch.cuda.empty_cache() # <-- prevents the crash\n\n# (2) synchronize each iteration\nfor i in range(50):\n a = torch", "environment": { "pytorch_version": "2.6.0", "cuda_version": "0.02" }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_185715", "source_url": "https://github.com/pytorch/pytorch/issues/185715", "source_type": "github_issue", "failure_type": "TORCH_COMPILE_DYNAMO", "title": "INTERNAL ASSERT FAILED: NYI SymInt equality in c10/core/Scalar.h when using torch.compile(dynamic=True) and torch.autograd.grad with torch.pow", "error_message": "nccl-cu13==2.29.7\n[pip3] nvidia-nvjitlink==13.0.88\n[pip3] nvidia-nvtx==13.0.85\n[pip3] torch==2.13.0.dev20260521+cu130\n[pip3] torchaudio==2.11.0.dev20260525+cu130\n[pip3] torchvision==0.28.0.dev20260525+cu130\n[pip3] triton==3.7.0+git88b227e2\n[conda] numpy 2.2.6 pypi_0 pypi\n[conda] nvidia-cublas 13.1.1.3 pypi_0 pypi\n[conda] nvidia-cuda-cupti 13.0.85 pypi_0 pypi\n[conda] nvidia-cuda-nvrtc 13.0.88 pypi_0 pypi\n[conda] nvidia-cuda-runtime 13.0.96 pypi_0 pypi\n[conda] nvidia-cudnn-cu13 9.20.0.48 pypi_0 py", "environment": { "pytorch_version": "2.13.0", "cuda_version": "13.0" }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 4426 }, { "failure_id": "lf_github_issue_185581", "source_url": "https://github.com/pytorch/pytorch/issues/185581", "source_type": "github_issue", "failure_type": "TORCH_COMPILE_DYNAMO", "title": "`torch.compile` fails at FakeTensor validation for ConvTranspose2d", "error_message": "Traceback (most recent call last):\n File \"/home/jason/Documents/DLCTestingv2/tests/test_re.py\", line 23, in \n compile_res = torch.compile(m)(x)\n ^^^^^^^^^^^^^^^^^^^\n File \"/home/jason/miniconda3/envs/dl/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py\", line 473, in __call__\n return super().__call__(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n...\n...\n File \"/home/jason/miniconda3/envs/dl/lib/python3.12/site-packages/torch/_dynamo/utils.py\", line 3740, in _wrap_graph_break_with_torch_runtime_err\n raise exc.with_traceback(e.__traceback__) from None\n File \"/home/jason/miniconda3/envs/dl/lib/python3.12/site-packages/torch/_dynamo/utils.py\", line 3737, in _wrap_graph_break_with_torch_runtime_err\n gb_fn()\n File \"/home/jason/miniconda3/envs/dl/lib/python3.12/site-packages/torch/_dynamo/utils.py\", line 3943, in \n lambda: unimplemented(\n ^^^^^^^^^^^^^^\n File \"/home/jason/miniconda3/envs/dl/lib/python3.12/site-packages/torch/_dynamo/exc.py\", line 653, in unimplemented\n raise Unsupported(\ntorch._dynamo.exc.TorchRuntimeError: RuntimeError when making fake tensor call\n Explanation: Dynamo failed to run FX node with fake tensors: call_function (*(FakeTensor(..., size=(2, 3, 2, 2)), Parameter(FakeTensor(..., size=(3, 1, 5, 5), requires_grad=True)), None, (1, 1), (3, 3), (0, 0), 1, (1, 1)), **{}): got RuntimeError('Given input size per channel: [2, 2]. Calculated output size per channel: [0, 0]. Output size is too small')\n Hint: Your code may result in an error when running in eager. Please double check that your code doesn't contain a similar error when actually running eager/uncompiled. You can do this by removing the `torch.compile` call, or by using `torch.compiler.set_stance(\"force_eager\")`. ", "environment": { "pytorch_version": "2.12.0", "cuda_version": "13.0" }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_185535", "source_url": "https://github.com/pytorch/pytorch/issues/185535", "source_type": "github_issue", "failure_type": "DEVICE_ASSERT", "title": "[inductor] cat lowering AssertionError with 1-D empty tensor and negative dim", "error_message": "### 🐛 Describe the bug\n\nInductor's lowering for `aten.cat.default` raises `AssertionError` when a 1-D empty tensor (e.g. `torch.tensor([])`) appears among the inputs and `dim` is negative with magnitude exceeding that tensor's rank.\n\nATen handles this via `cat_should_skip_tensor`: 1-D tensors with numel==0 are skipped during dimension validation and concatenation. Inductor's `cat` lowering does not replicate this skip, so `_validate_dim` always uses `inputs[0]` as the reference for normalizing n", "environment": {}, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 2723 }, { "failure_id": "lf_github_issue_185513", "source_url": "https://github.com/pytorch/pytorch/issues/185513", "source_type": "github_issue", "failure_type": "TORCH_COMPILE_INDUCTOR", "title": "[TMA] NameError in `sum` when TMA enabled", "error_message": "### 🐛 Describe the bug\n\n### Repro\n```\nimport torch\nimport torch._inductor.config as inductor_config\nfrom torch.testing._internal.common_utils import run_tests, TestCase\n\n\nclass TestTMAComboKernelNameError(TestCase):\n @inductor_config.patch(\n {\n \"triton.use_tensor_descriptor\": True,\n \"assume_aligned_inputs\": True,\n \"combo_kernels\": True,\n \"combo_kernel_per_subkernel_blocks\": False,\n }\n )\n def test_combo_kernel_per_subkernel_rblock", "environment": {}, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_185512", "source_url": "https://github.com/pytorch/pytorch/issues/185512", "source_type": "github_issue", "failure_type": "VERSION_MISMATCH", "title": "RuntimeError: CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH in F.conv2d on PyTorch nightly (cu130)", "error_message": "Traceback (most recent call last):\n File \"/tmp/bug.py\", line 31, in \n main()\n File \"/tmp/bug.py\", line 23, in main\n out = F.conv2d(x, weight, bias, padding=1)\nRuntimeError: CUDNN_BACKEND_TENSOR_DESCRIPTOR cudnnFinalize failedptrDesc->finalize() cudnn_status: CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH\n```", "environment": { "pytorch_version": "2.13.0", "cuda_version": "2.13" }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_185497", "source_url": "https://github.com/pytorch/pytorch/issues/185497", "source_type": "github_issue", "failure_type": "GRADIENT_EXPLOSION", "title": "[pt2] RuntimeError: variable modified by inplace operation during backward in compiled mode (succeeds in eager)", "error_message": "Traceback (most recent call last):\n File \"/tmp/bug.py\", line 29, in run\n loss.backward()\n File \"/home/xyt19/miniconda3/envs/torch-nightly/lib/python3.10/site-packages/torch/_tensor.py\", line 633, in backward\n torch.autograd.backward(\n File \"/home/xyt19/miniconda3/envs/torch-nightly/lib/python3.10/site-packages/torch/autograd/__init__.py\", line 395, in backward\n _engine_run_backward(\n File \"/home/xyt19/miniconda3/envs/torch-nightly/lib/python3.10/site-packages/torch/autograd/graph.py\", line 913, in _engine_run_backward\n return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass\n File \"/home/xyt19/miniconda3/envs/torch-nightly/lib/python3.10/site-packages/torch/autograd/function.py\", line 333, in apply_boxed\n return self._get_user_fn()(self, *args)\n File \"/home/xyt19/miniconda3/envs/torch-nightly/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py\", line 3454, in backward\n return CompiledFunction._bwd_fn(\n File \"/home/xyt19/miniconda3/envs/torch-nightly/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/subclass_codegen.py:codegen(compiled_function_backward)\", line 10, in _compiled_backward\nRuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [3, 128, 128]] is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True, check_nan=False).\n```", "environment": { "pytorch_version": "2.13.0", "cuda_version": "0.5" }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_185494", "source_url": "https://github.com/pytorch/pytorch/issues/185494", "source_type": "github_issue", "failure_type": "TORCH_COMPILE_DYNAMO", "title": "[dynamo] LazyBatchNorm2d fails under torch.compile(dynamic=True) — \"SymIntArrayRef expected to contain only concrete integers\"", "error_message": "Traceback (most recent call last):\n ...\n File \"torch/_dynamo/variables/nn_module.py\", line 118, in initialize_lazy_module\n mod._infer_parameters(mod, fake_args, fake_kwargs)\n File \"torch/nn/modules/lazy.py\", line 263, in _infer_parameters\n module.initialize_parameters(*args, **kwargs)\n File \"torch/nn/modules/batchnorm.py\", line 289, in initialize_parameters\n self.weight.materialize((self.num_features,))\n File \"torch/nn/parameter.py\", line 147, in materialize\n self.data = torch.empty(shape, device=device, dtype=dtype)\ntorch._dynamo.exc.InternalTorchDynamoError: RuntimeError: /__w/pytorch/pytorch/build/aten/src/ATen/RegisterCPU_1.cpp:2519: SymIntArrayRef expected to contain only concrete integers", "environment": { "pytorch_version": "2.13.0", "cuda_version": "12.6" }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 2472 }, { "failure_id": "lf_github_issue_185488", "source_url": "https://github.com/pytorch/pytorch/issues/185488", "source_type": "github_issue", "failure_type": "TORCH_COMPILE_DYNAMO", "title": "[dynamo] LazyBatchNorm2d fails under torch.compile(dynamic=True) — \"SymIntArrayRef expected to contain only concrete integers\"", "error_message": "Traceback (most recent call last):\n ...\n File \"torch/_dynamo/variables/nn_module.py\", line 118, in initialize_lazy_module\n mod._infer_parameters(mod, fake_args, fake_kwargs)\n File \"torch/nn/modules/lazy.py\", line 263, in _infer_parameters\n module.initialize_parameters(*args, **kwargs)\n File \"torch/nn/modules/batchnorm.py\", line 289, in initialize_parameters\n self.weight.materialize((self.num_features,))\n File \"torch/nn/parameter.py\", line 147, in materialize\n self.data = torch.empty(shape, device=device, dtype=dtype)\ntorch._dynamo.exc.InternalTorchDynamoError: RuntimeError: /__w/pytorch/pytorch/build/aten/src/ATen/RegisterCPU_1.cpp:2519: SymIntArrayRef expected to contain only concrete integers", "environment": { "pytorch_version": "2.13.0", "cuda_version": "12.6" }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 2269 }, { "failure_id": "lf_github_issue_186408", "source_url": "https://github.com/pytorch/pytorch/issues/186408", "source_type": "github_issue", "failure_type": "TORCH_COMPILE_DYNAMO", "title": "torch.compile fails tracing platform.machine() on Python 3.12", "error_message": "### 🐛 Describe the bug\n\n`torch.compile(fullgraph=True)` fails while tracing `platform.machine()` on Python 3.12. The same call works in eager mode.\n\nThis repro uses `backend=\"eager\"`, so the failure appears to be in Dynamo tracing rather than Inductor or a device backend.\n\n```python\nimport platform\nimport traceback\n\nimport torch\n\nprint(\"torch:\", torch.__version__)\nprint(\"python:\", platform.python_version())\nprint(\"eager platform.machine():\", platform.machine())\n\n@torch.compile(backend=\"eager\", f", "environment": {}, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 1723 }, { "failure_id": "lf_github_issue_186369", "source_url": "https://github.com/pytorch/pytorch/issues/186369", "source_type": "github_issue", "failure_type": "TORCH_COMPILE_INDUCTOR", "title": "[MPS][Inductor] Metal codegen emits self-referential auto (`tmp_scoped_0 = static_cast(tmp_scoped_0)`) from nested masked-index scopes", "error_message": "### 🐛 Describe the bug\n\nOn MPS, `torch.compile` (Inductor backend) generates **invalid Metal source** for a kernel that contains *nested* masked-index sub-blocks. The inner block restarts the `tmp_scoped_N` temporary-name counter from `0`, re-declaring names that are still live in the enclosing scope. One of those re-declarations reads the name it is declaring, producing a self-referential `auto`:\n\n```cpp\nauto tmp_scoped_0 = static_cast(tmp_scoped_0);\n```\n\nThe Metal compiler rejects it:\n\n``", "environment": {}, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_issue_186357", "source_url": "https://github.com/pytorch/pytorch/issues/186357", "source_type": "github_issue", "failure_type": "TORCH_COMPILE_AOTI", "title": "AOTInductor on Windows: models with >2GB constants fail to load — 32-bit `lseek` overflow (`weights_offset must be aligned to 16K boundary`)", "error_message": "RuntimeError: weights_offset must be aligned to 16K boundary` (or, depending on the\nlow-bit luck of the corrupted offset, loads and then hits an illegal memory access at run). The\nweights are in fact laid out at a correctly 16K-aligned file offset — the runtime just **mis-reads\nthe file size** because `model_base.h` uses 32-bit `lseek`, which overflows for files larger than\n2 GB on Windows/MSVC.\n\nThis is distinct from #145610 (ARM 64K-page granularity); same assertion message, different cause.\n\n", "environment": { "cuda_version": "12.8", "gpu_type": "RTX 5060" }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 3644 }, { "failure_id": "lf_github_issue_186216", "source_url": "https://github.com/pytorch/pytorch/issues/186216", "source_type": "github_issue", "failure_type": "TORCH_COMPILE_AOTI", "title": "Bug ，Cannot compile Native C++ API to export LibTorch trained models directly to AOTInductor format", "error_message": "### 🐛 Describe the bug\n\n```\n\n### Versions\n\nWe are requesting a supported pathway or native C++ API to export models trained purely in C++ (LibTorch) directly to the AOTInductor (.pt2) format, allowing them to be loaded via torch::inductor::AOTIModelPackageLoader without requiring a Python-based graph recreation step.\n\nMotivation\nIn our engineering stack, we have invested heavily in pure C++ LibTorch pipelines for both training and inference. We manage complex native memory environments, strict m", "environment": {}, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 3023 }, { "failure_id": "lf_github_issue_186213", "source_url": "https://github.com/pytorch/pytorch/issues/186213", "source_type": "github_issue", "failure_type": "OOM_FRAGMENTATION", "title": "CUDA expandable IPC receiver reserves ~9/8 of total GPU memory VA per imported handle and can exhaust device virtual address space", "error_message": "torch.cuda.memory_allocated()` and `torch.cuda.memory_reserved()` stay at `0` in the receiver, and `mem_get_info()` still reports free memory.\n\nObserved failure:\n\n```text\nRuntimeError: CUDA driver error: invalid argument\n```\n\nWith local instrumentation around `getIpcDevPtr` and `ExpandableSegment::fromShared`, I observed the failure at:\n\n```text\n[CUDA IPC cache] miss-begin hits=0 misses=2621 inserts=2620 erases=0 failures=0 cache_size=2620 handle_size=30 device=0\n[CUDA IPC expandable] fromShared", "environment": {}, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 5000 }, { "failure_id": "lf_github_search_16668", "source_url": "https://github.com/pytorch/pytorch/issues/16668", "source_type": "github_search", "failure_type": "OOM_FRAGMENTATION", "title": "Reduce fragmentation with CUDA caching allocator when using many streams", "error_message": "At the moment, once we retrieve a block from `cudaMalloc` for the CUDA caching allocator, it is permanently associated with whatever stream was current at the time it was allocated. Even if it is subsequently split, all splits of the block live on the same stream. This means that if you have a program which uses many streams, you will have a much greater amount of fragmentation, scaled with the number of streams you use. On some internal Caffe2 video processing workflows, we have noticed that th", "environment": {}, "fix": null, "verified_fix": false, "framework": "Unknown", "confidence": 0.6, "is_cascade": false, "raw_body_length": 1100 }, { "failure_id": "lf_github_search_171771", "source_url": "https://github.com/pytorch/pytorch/issues/171771", "source_type": "github_search", "failure_type": "STRAGGLER", "title": "[RFC] Optimize CUDA Allocator's Synchronization Behavior During OOM", "error_message": "### 🐛 Describe the current behavior\nWhen the PyTorch CUDA caching allocator fails to allocate GPU memory, it falls back to a strategy of waiting for **all** memory blocks marked with `record_stream` to be freed (by waiting for all associated CUDA events to complete), then reclaims these blocks to satisfy the current allocation request.\n\nThis behavior has two critical issues for production workloads:\n1. **Inefficient wait logic**: The allocator waits for all blocks to be freed even if only a subs", "environment": {}, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 3122 }, { "failure_id": "lf_github_search_173049", "source_url": "https://github.com/pytorch/pytorch/issues/173049", "source_type": "github_search", "failure_type": "OOM_FRAGMENTATION", "title": "OOM error suggests expandable_segments even when enabled", "error_message": "When a CUDA OOM error occurs, PyTorch suggests trying `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` to avoid fragmentation.\nHowever, this message appears currently even if `expandable_segments` is already enabled by the user. This can be confusing.\n\nIt would be clearer if we checked `CUDAAllocatorConfig::expandable_segments()` before showing this tip, so it only appears when the feature is actually disabled.\n\ncc @ptrblck @msaroufim @eqy @jerryzh168 @tinglvv @nWEIdia @malfet", "environment": {}, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 483 }, { "failure_id": "lf_github_search_186213", "source_url": "https://github.com/pytorch/pytorch/issues/186213", "source_type": "github_search", "failure_type": "OOM_FRAGMENTATION", "title": "CUDA expandable IPC receiver reserves ~9/8 of total GPU memory VA per imported handle and can exhaust device virtual address space", "error_message": "Traceback (most recent call last):\n File \"/workspace/ipc_oom.py\", line 104, in consumer\n tensor = torch.multiprocessing.reductions.rebuild_cuda_tensor(*metadata)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \".../torch/multiprocessing/reductions.py\", line 181, in rebuild_cuda_tensor\n storage = storage_cls._new_shared_cuda(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \".../torch/storage.py\", line 1457, in _new_shared_cuda\n return torch.UntypedStorage._new_shared_cuda(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nRuntimeError: CUDA driver error: invalid argument\n```", "environment": { "pytorch_version": "2.9.0", "cuda_version": "13.0" }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 14929 }, { "failure_id": "lf_github_search_111363", "source_url": "https://github.com/pytorch/pytorch/issues/111363", "source_type": "github_search", "failure_type": "OOM_FRAGMENTATION", "title": "Simulating lower memory on GPU does not indicate simulated memory in error message", "error_message": "torch.cuda.set_per_process_memory_fraction(0.5)` API which I'm using to simulate lower CUDA memory conditions. \r\n\r\nHowever, upon an OOM I get an error message such as\r\n\r\n```\r\n content_understanding.utils.injected_exception.NonRetryableExceptionWrapper: (OutOfMemoryError) CUDA out of memory. Tried to allocate 64.00 MiB. GPU 0 has a total capacty of 79.15 GiB of which 38.09 GiB is free. Including non-PyTorch memory, this process has 41.06 GiB memory in use. Of the allocated memory 39.10 GiB is al", "environment": { "cuda_version": "0.5", "gpu_type": "A100" }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 1027 }, { "failure_id": "lf_github_search_104875", "source_url": "https://github.com/pytorch/pytorch/issues/104875", "source_type": "github_search", "failure_type": "OOM_FRAGMENTATION", "title": "torch/testing/_comparison.py: If you are a user and see this message during normal operation please file an issue", "error_message": "Traceback (most recent call last):\r\n File \"/foo/localenv/lib/python3.9/site-packages/torch/testing/_comparison.py\", line 1224, in not_close_error_metas\r\n pair.compare()\r\n File \"/foo/localenv/lib/python3.9/site-packages/torch/testing/_comparison.py\", line 706, in compare\r\n self._compare_values(actual, expected)\r\n File \"/foo/localenv/lib/python3.9/site-packages/torch/testing/_comparison.py\", line 824, in _compare_values\r\n compare_fn(\r\n File \"/foo/lib/python3.9/site-packages/torch/testing/_comparison.py\", line 994, in _compare_regular_values_close\r\n matches = torch.isclose(\r\ntorch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 1.95 GiB total capacity; 1.27 GiB already allocated; 180.38 MiB free; 1.27 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF\r\n\r\nThe above exception was the direct cause of the following exception:\r\n\r\nTraceback (most recent call last):\r\n File \"mwe.py\", line 26, in \r\n torch.testing.assert_close(yf, ys)\r\n File \"/foo/localenv/lib/python3.9/site-packages/torch/testing/_comparison.py\", line 1489, in assert_close\r\n error_metas = not_close_error_metas(\r\n File \"/foo/localenv/lib/python3.9/site-packages/torch/testing/_comparison.py\", line 1230, in not_close_error_metas\r\n raise RuntimeError(\r\nRuntimeError: Comparing\r\n\r\nTensorLikePair(\r\n id=(),\r\n actual=tensor([[0.5528+0.6277j, 0.2594+0.9218j, 0.6938+0.6858j, ...,\r\n 0.3728+0.2267j, 0.9894+0.9470j, 0.1317+0.7768j],\r\n [0.6751+0.5199j, 0.6546+0.8712j, 0.7528+0.3251j, ...,\r\n 0.7132+0.0744j, 0.5763+0.7044j, 0.4192+0.1781j],\r\n [0.9773+0.2660j, 0.0375+0.5843j, 0.8705+0.7881j, ...,\r\n 0.4815+0.1623j, 0.9864+0.8712j, 0.6572+0.1675j],\r\n ...,\r\n [0.9890+0.5754j, 0.4324+0.9647j, 0.1394+0.7539j, ...,\r\n 0.3246+0.4463j, 0.5527+0.6973j, 0.0100", "environment": { "pytorch_version": "2.0.1", "cuda_version": "256.00" }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 8037 }, { "failure_id": "lf_github_search_171771", "source_url": "https://github.com/pytorch/pytorch/issues/171771", "source_type": "github_search", "failure_type": "STRAGGLER", "title": "[RFC] Optimize CUDA Allocator's Synchronization Behavior During OOM", "error_message": "### 🐛 Describe the current behavior\nWhen the PyTorch CUDA caching allocator fails to allocate GPU memory, it falls back to a strategy of waiting for **all** memory blocks marked with `record_stream` to be freed (by waiting for all associated CUDA events to complete), then reclaims these blocks to satisfy the current allocation request.\n\nThis behavior has two critical issues for production workloads:\n1. **Inefficient wait logic**: The allocator waits for all blocks to be freed even if only a subs", "environment": {}, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 3122 }, { "failure_id": "lf_github_search_173049", "source_url": "https://github.com/pytorch/pytorch/issues/173049", "source_type": "github_search", "failure_type": "OOM_FRAGMENTATION", "title": "OOM error suggests expandable_segments even when enabled", "error_message": "When a CUDA OOM error occurs, PyTorch suggests trying `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` to avoid fragmentation.\nHowever, this message appears currently even if `expandable_segments` is already enabled by the user. This can be confusing.\n\nIt would be clearer if we checked `CUDAAllocatorConfig::expandable_segments()` before showing this tip, so it only appears when the feature is actually disabled.\n\ncc @ptrblck @msaroufim @eqy @jerryzh168 @tinglvv @nWEIdia @malfet", "environment": {}, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 483 }, { "failure_id": "lf_github_search_186213", "source_url": "https://github.com/pytorch/pytorch/issues/186213", "source_type": "github_search", "failure_type": "OOM_FRAGMENTATION", "title": "CUDA expandable IPC receiver reserves ~9/8 of total GPU memory VA per imported handle and can exhaust device virtual address space", "error_message": "Traceback (most recent call last):\n File \"/workspace/ipc_oom.py\", line 104, in consumer\n tensor = torch.multiprocessing.reductions.rebuild_cuda_tensor(*metadata)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \".../torch/multiprocessing/reductions.py\", line 181, in rebuild_cuda_tensor\n storage = storage_cls._new_shared_cuda(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \".../torch/storage.py\", line 1457, in _new_shared_cuda\n return torch.UntypedStorage._new_shared_cuda(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nRuntimeError: CUDA driver error: invalid argument\n```", "environment": { "pytorch_version": "2.9.0", "cuda_version": "13.0" }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 14929 }, { "failure_id": "lf_github_search_111363", "source_url": "https://github.com/pytorch/pytorch/issues/111363", "source_type": "github_search", "failure_type": "OOM_FRAGMENTATION", "title": "Simulating lower memory on GPU does not indicate simulated memory in error message", "error_message": "torch.cuda.set_per_process_memory_fraction(0.5)` API which I'm using to simulate lower CUDA memory conditions. \r\n\r\nHowever, upon an OOM I get an error message such as\r\n\r\n```\r\n content_understanding.utils.injected_exception.NonRetryableExceptionWrapper: (OutOfMemoryError) CUDA out of memory. Tried to allocate 64.00 MiB. GPU 0 has a total capacty of 79.15 GiB of which 38.09 GiB is free. Including non-PyTorch memory, this process has 41.06 GiB memory in use. Of the allocated memory 39.10 GiB is al", "environment": { "cuda_version": "0.5", "gpu_type": "A100" }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 1027 }, { "failure_id": "lf_github_search_111646", "source_url": "https://github.com/pytorch/pytorch/issues/111646", "source_type": "github_search", "failure_type": "SILENT_HANG", "title": "torchrun: elastic training not restarted on missing keep-alive heartbeat/scale-down event", "error_message": "NCCL operations and wait by default 30 minutes until they finish with a timeout error.\r\n\r\n*Expected Behaviour*\r\nIf a worker misses its heartbeat/leaves the rendevous, a new rendevous should happen.\r\n\r\n*Minimal Example*\r\n\r\nThere are two scripts for the master worker and a faulty worker\r\n\r\n**master**\r\n```python\r\nimport torch.distributed.run\r\nimport logging\r\nlogging.getLogger(\"torch.distributed.elastic.rendezvous.dynamic_rendezvous\").level=10\r\ntorch.distributed.run.main([\"--nnodes=1:4\",\"--rdzv-back", "environment": { "pytorch_version": "2.1.0", "cuda_version": "11.8", "gpu_type": "RTX 3080" }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": true, "raw_body_length": 13499 }, { "failure_id": "lf_github_search_139884", "source_url": "https://github.com/pytorch/pytorch/issues/139884", "source_type": "github_search", "failure_type": "DEVICE_ASSERT", "title": "Persistent memory leak from failed pinned memory allocation", "error_message": "Traceback (most recent call last):\r\n File \"[...]/leakmem.py\", line 2, in \r\n torch.empty((1024,1024,1024), dtype=torch.float32, pin_memory=True)\r\nRuntimeError: CUDA error: invalid argument\r\nCUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\r\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1\r\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\r\n```\r\n\r\nThe failure happens when the allocated size is sufficiently large. That's alright, I understand it has to do with limitations with pinned memory. Not so fine is that 4 gigs of memory apparently does get consumed and stays that way after the process exits, and even after the user has no processes. I don't see anything relevant in `/dev/shm/`. I don't know how to free it other than by rebooting.\r\n\r\n### Versions\r\n\r\n#### Affected system\r\n\r\n```\r\nCollecting environment information...\r\nPyTorch version: 2.5.1+cu124\r\nIs debug build: False\r\nCUDA used to build PyTorch: 12.4\r\nROCM used to build PyTorch: N/A\r\n\r\nOS: openSUSE Tumbleweed (x86_64)\r\nGCC version: (SUSE Linux) 14.2.1 20241007 [revision 4af44f2cf7d281f3e4f3957efce10e8b2ccb2ad3]\r\nClang version: Could not collect\r\nCMake version: version 3.30.5\r\nLibc version: glibc-2.40\r\n\r\nPython version: 3.11.10 (main, Sep 09 2024, 17:03:08) [GCC] (64-bit runtime)\r\nPython platform: Linux-6.11.5-2-default-x86_64-with-glibc2.40\r\nIs CUDA available: True\r\nCUDA runtime version: Could not collect\r\nCUDA_MODULE_LOADING set to: LAZY\r\nGPU models and configuration: GPU 0: NVIDIA GeForce RTX 4070\r\nNvidia driver version: 550.127.05\r\ncuDNN version: Could not collect\r\nHIP runtime version: N/A\r\nMIOpen runtime version: N/A\r\nIs XNNPACK available: True\r\n\r\nCPU:\r\nArchitecture: x86_64\r\nCPU op-mode(s): 32-bit, 64-bit\r\nAddress sizes: 48 bits physical, 48 bits virtual\r\nByte Order: Little Endian\r\nCPU(s): ", "environment": { "pytorch_version": "2.5.1", "cuda_version": "12.4", "gpu_type": "RTX 4070" }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 10403 }, { "failure_id": "lf_github_search_111646", "source_url": "https://github.com/pytorch/pytorch/issues/111646", "source_type": "github_search", "failure_type": "SILENT_HANG", "title": "torchrun: elastic training not restarted on missing keep-alive heartbeat/scale-down event", "error_message": "NCCL operations and wait by default 30 minutes until they finish with a timeout error.\r\n\r\n*Expected Behaviour*\r\nIf a worker misses its heartbeat/leaves the rendevous, a new rendevous should happen.\r\n\r\n*Minimal Example*\r\n\r\nThere are two scripts for the master worker and a faulty worker\r\n\r\n**master**\r\n```python\r\nimport torch.distributed.run\r\nimport logging\r\nlogging.getLogger(\"torch.distributed.elastic.rendezvous.dynamic_rendezvous\").level=10\r\ntorch.distributed.run.main([\"--nnodes=1:4\",\"--rdzv-back", "environment": { "pytorch_version": "2.1.0", "cuda_version": "11.8", "gpu_type": "RTX 3080" }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": true, "raw_body_length": 13499 }, { "failure_id": "lf_github_search_160169", "source_url": "https://github.com/pytorch/pytorch/issues/160169", "source_type": "github_search", "failure_type": "DDP_ERROR", "title": "When using DP + TP, DP only parameters diverge across TP ranks if using operations with non-deterministic implementations", "error_message": "Traceback (most recent call last):\n[rank0]: File \"/home/user02107/vlm-train/tp_with_dp_only.py\", line 194, in \n[rank0]: main()\n[rank0]: File \"/home/user02107/vlm-train/tp_with_dp_only.py\", line 183, in main\n[rank0]: torch.testing.assert_close(\n[rank0]: File \"/usr/local/lib/python3.12/dist-packages/torch/testing/_comparison.py\", line 1519, in assert_close\n[rank0]: raise error_metas[0].to_error(msg)\n[rank0]: AssertionError: [0, 0]: dp2.weight\n[rank0]: Tensor-likes are not close!", "environment": { "pytorch_version": "2.8.0", "cuda_version": "12.9", "gpu_type": "H100", "num_gpus": 4 }, "fix": null, "verified_fix": false, "framework": "FSDP", "confidence": 0.6, "is_cascade": true, "raw_body_length": 19315 }, { "failure_id": "lf_github_search_31356", "source_url": "https://github.com/pytorch/pytorch/issues/31356", "source_type": "github_search", "failure_type": "OOM_FRAGMENTATION", "title": "Optimizing DLRM for CPU", "error_message": "## 🚀 Feature\r\nA number of optimization and performance tuning for DLRM on CPU\r\n\r\n## Motivation\r\n\r\nRecommendation systems are one of the most common DL workloads in the cloud or enterprise server room. Very often the recommendation system burns most compute cycles in the data center among all DL workload. DLRM is a state-of-the-art deep learning recommendation model which is composed of compute intensive MLP layers and memory intensive and capacity limited embedding layers. Due to the memory ca", "environment": {}, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 4614 }, { "failure_id": "lf_github_search_139884", "source_url": "https://github.com/pytorch/pytorch/issues/139884", "source_type": "github_search", "failure_type": "DEVICE_ASSERT", "title": "Persistent memory leak from failed pinned memory allocation", "error_message": "Traceback (most recent call last):\r\n File \"[...]/leakmem.py\", line 2, in \r\n torch.empty((1024,1024,1024), dtype=torch.float32, pin_memory=True)\r\nRuntimeError: CUDA error: invalid argument\r\nCUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\r\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1\r\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\r\n```\r\n\r\nThe failure happens when the allocated size is sufficiently large. That's alright, I understand it has to do with limitations with pinned memory. Not so fine is that 4 gigs of memory apparently does get consumed and stays that way after the process exits, and even after the user has no processes. I don't see anything relevant in `/dev/shm/`. I don't know how to free it other than by rebooting.\r\n\r\n### Versions\r\n\r\n#### Affected system\r\n\r\n```\r\nCollecting environment information...\r\nPyTorch version: 2.5.1+cu124\r\nIs debug build: False\r\nCUDA used to build PyTorch: 12.4\r\nROCM used to build PyTorch: N/A\r\n\r\nOS: openSUSE Tumbleweed (x86_64)\r\nGCC version: (SUSE Linux) 14.2.1 20241007 [revision 4af44f2cf7d281f3e4f3957efce10e8b2ccb2ad3]\r\nClang version: Could not collect\r\nCMake version: version 3.30.5\r\nLibc version: glibc-2.40\r\n\r\nPython version: 3.11.10 (main, Sep 09 2024, 17:03:08) [GCC] (64-bit runtime)\r\nPython platform: Linux-6.11.5-2-default-x86_64-with-glibc2.40\r\nIs CUDA available: True\r\nCUDA runtime version: Could not collect\r\nCUDA_MODULE_LOADING set to: LAZY\r\nGPU models and configuration: GPU 0: NVIDIA GeForce RTX 4070\r\nNvidia driver version: 550.127.05\r\ncuDNN version: Could not collect\r\nHIP runtime version: N/A\r\nMIOpen runtime version: N/A\r\nIs XNNPACK available: True\r\n\r\nCPU:\r\nArchitecture: x86_64\r\nCPU op-mode(s): 32-bit, 64-bit\r\nAddress sizes: 48 bits physical, 48 bits virtual\r\nByte Order: Little Endian\r\nCPU(s): ", "environment": { "pytorch_version": "2.5.1", "cuda_version": "12.4", "gpu_type": "RTX 4070" }, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 10403 }, { "failure_id": "lf_github_search_31356", "source_url": "https://github.com/pytorch/pytorch/issues/31356", "source_type": "github_search", "failure_type": "OOM_FRAGMENTATION", "title": "Optimizing DLRM for CPU", "error_message": "## 🚀 Feature\r\nA number of optimization and performance tuning for DLRM on CPU\r\n\r\n## Motivation\r\n\r\nRecommendation systems are one of the most common DL workloads in the cloud or enterprise server room. Very often the recommendation system burns most compute cycles in the data center among all DL workload. DLRM is a state-of-the-art deep learning recommendation model which is composed of compute intensive MLP layers and memory intensive and capacity limited embedding layers. Due to the memory ca", "environment": {}, "fix": null, "verified_fix": false, "framework": "PyTorch", "confidence": 0.6, "is_cascade": false, "raw_body_length": 4614 }, { "failure_id": "lf_github_search_96629", "source_url": "https://github.com/pytorch/pytorch/issues/96629", "source_type": "github_search", "failure_type": "SILENT_HANG", "title": "Dataloader should kill & restart workers when timeout is hit", "error_message": "### 🚀 The feature, motivation and pitch\n\nWhen using `timeout`, instead of crashing when the timeout is hit, the Dataloader should instead kill and restart problematic workers. Ideally, the worker should also be able to report the stack frame it is stuck on when being killed. This would be extremely useful for debugging code that works when `num_workers=0` but doesn't when `num_workers>0`. It also can save quite a bit of frustration when training hangs.\n\n### Alternatives\n\n_No response_\n\n### Addit", "environment": {}, "fix": null, "verified_fix": false, "framework": "Unknown", "confidence": 0.6, "is_cascade": false, "raw_body_length": 926 }, { "failure_id": "lf_github_search_14766", "source_url": "https://github.com/huggingface/transformers/issues/14766", "source_type": "github_search", "failure_type": "NAN_LOSS", "title": "Nan when training LayoutLM_V2 Model", "error_message": "## Environment info\r\n\r\n\r\n- `transformers` version: 4.13.0\r\n- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic\r\n- Python version: 3.7.12\r\n- PyTorch version (GPU) : 1.10.0+cu111\r\n- Tensorflow version (GPU): 2.7.0\r\n- Flax version: not installed\r\n- Jax version: not installed\r\n- JaxLib version: not installed\r\n\r\n\r\n### Who can help\r\n\r\n\r\n- `transformers` version: 4.13.0\r\n- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic\r\n- Python version: 3.7.12\r\n- PyTorch version (GPU) : 1.10.0+cu111\r\n- Tensorflow version (GPU): 2.7.0\r\n- Flax version: not installed\r\n- Jax version: not installed\r\n- JaxLib version: not installed\r\n\r\n\r\n### Who can help\r\n\r\n\r\n- `transformers` version: 4.13.0\r\n- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic\r\n- Python version: 3.7.12\r\n- PyTorch version (GPU) : 1.10.0+cu111\r\n- Tensorflow version (GPU): 2.7.0\r\n- Flax version: not installed\r\n- Jax version: not installed\r\n- JaxLib version: not installed\r\n\r\n\r\n### Who can help\r\n