Qwen3.5 fa2 微调报错 illegal memory
bugpending
### Reminder
- [x] I have read the above rules and searched the existing issues.
### System Info
LlamaFactory 版本: 0.9.5dev0
Transformers 版本: transformers: 5.2.0
Python 版本: 3.12
causal-conv1d: 1.6.1
flash-linear-attention: 0.5.0
在以下三种环境组合下测试均相同报错 illegal memory
env1:
```
torch: 2.7.0+cu128
flash-attention: 2.7.4.post1
```
env2:
```
torch: 2.9.1+cu128
flash-attention: 2.8.3+cu128torch2.9
```
env3:
```
torch: 2.11.0+cu128
flash-attention: 2.8.3+cu128torch2.11
```
完整报错日志：
```
0%|
| 0/375 [00:00<?, ?it/s][rank3]: Traceback (most recent call last):
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/random.py", line 213, in fork_rng
[rank3]: yield
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/utils/checkpoint.py", line 308, in backward
[rank3]: outputs = ctx.run_function(*detached_inputs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
[rank3]: return self._call_impl(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1885, in _call_impl
[rank3]: return inner()
[rank3]: ^^^^^^^
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1833, in inner
[rank3]: result = forward_call(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/workspace/LlamaFactory/src/llamafactory/model/patcher.py", line 139, in _patched_decoder_forward
[rank3]: hidden_states = self.mlp(hidden_states)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
[rank3]: return self._call_impl(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1885, in _call_impl
[rank3]: return inner()
[rank3]: ^^^^^^^
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1833, in inner
[rank3]: result = forward_call(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/transformers/models/qwen3_5/modeling_qwen3_5.py", line 803, in forward
[rank3]: down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
[rank3]: ^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
[rank3]: return self._call_impl(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1885, in _call_impl
[rank3]: return inner()
[rank3]: ^^^^^^^
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1822, in inner
[rank3]: args_result = hook(self, args)
[rank3]: ^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1263, in _fn
[rank3]: return fn(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/deepspeed/runtime/zero/parameter_offload.py", line 300, in _pre_forward_module_hook
[rank3]: self.pre_sub_module_forward_function(module)
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
[rank3]: return func(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/deepspeed/runtime/zero/parameter_offload.py", line 475, in pre_sub_module_forward_function
[rank3]: param_coordinator.fetch_sub_module(sub_module, forward=True)
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1263, in _fn
[rank3]: return fn(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/deepspeed/utils/nvtx.py", line 20, in wrapped_fn
[rank3]: ret_val = func(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
[rank3]: return func(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/deepspeed/runtime/zero/partitioned_param_coordinator.py", line 335, in fetch_sub_module
[rank3]: self.__ongoing_fetch_events.popleft().synchronize()
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/cuda/streams.py", line 254, in synchronize
[rank3]: super().synchronize()
[rank3]: torch.AcceleratorError: CUDA error: an illegal memory access was encountered
[rank3]: Search for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
[rank3]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[rank3]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1
[rank3]: Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
[rank3]: During handling of the above exception, another exception occurred:
[rank3]: Traceback (most recent call last):
[rank3]: File "/workspace/LlamaFactory/src/llamafactory/launcher.py", line 185, in <module>
[rank3]: run_exp()
[rank3]: File "/workspace/LlamaFactory/src/llamafactory/train/tuner.py", line 137, in run_exp
[rank3]: _training_function(config={"args": args, "callbacks": callbacks})
[rank3]: File "/workspace/LlamaFactory/src/llamafactory/train/tuner.py", line 105, in _training_function
[rank3]: run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
[rank3]: File "/workspace/LlamaFactory/src/llamafactory/train/sft/workflow.py", line 140, in run_sft
[rank3]: train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/transformers/trainer.py", line 1412, in train
[rank3]: return inner_training_loop(
[rank3]: ^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/transformers/trainer.py", line 1742, in _inner_training_loop
[rank3]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/transformers/trainer.py", line 1979, in training_step
[rank3]: self.accelerator.backward(loss, **kwargs)
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/accelerate/accelerator.py", line 2732, in backward
[rank3]: self.deepspeed_engine_wrapped.backward(loss, sync_gradients=self.sync_gradients, **kwargs)
[rank3]: File "/root/miniconda3/envs/lmf2/lib/python3.12/site-packages/accelerate/utils/deepspeed.py", line 270, in backward
[rank3]: self.engine.backward(loss, **kwargs
......
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x7f195a37205d in /root/miniconda3[199/1904]
/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xc114 (0x7f1998173114 in /root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x50 (0x7f1885d30ef0 in /root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/lib/libtorch_
cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x68 (0x7f1885d3fa08 in /root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x9c3 (0x7f1885d443b3 in /root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x157 (0x7f1885d46597 in /root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xdc253 (0x7f1959eb0253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #7: <unknown function> + 0x94ac3 (0x7f1998e98ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #8: clone + 0x44 (0x7f1998f29a04 in /lib/x86_64-linux-gnu/libc.so.6)
terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG ID 0 PG GUID 0(default_pg) Rank 3] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
Search for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Exception raised from query at /pytorch/c10/cuda/CUDAEvent.h:111 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x7f195a37205d in /root/miniconda3/envs/lmf2
/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xc114 (0x7f1998173114 in /root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x50 (0x7f1885d30ef0 in /root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/lib/libtorch_
cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x68 (0x7f1885d3fa08 in /root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x9c3 (0x7f1885d443b3 in /root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x157 (0x7f1885d46597 in /root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xdc253 (0x7f1959eb0253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #7: <unknown function> + 0x94ac3 (0x7f1998e98ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #8: clone + 0x44 (0x7f1998f29a04 in /lib/x86_64-linux-gnu/libc.so.6)
Exception raised from run at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:2125 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x7f195a37205d in /root/miniconda3/envs/lmf2
/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x9b5d75 (0x7f18855a6d75 in /root/miniconda3/envs/lmf2/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0xdc253 (0x7f1959eb0253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #3: <unknown function> + 0x94ac3 (0x7f1998e98ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #4: clone + 0x44 (0x7f1998f29a04 in /lib/x86_64-linux-gnu/libc.so.6)
W0427 14:41:01.613000 49366 site-packages/torch/distributed/elastic/multiprocessing/api.py:1012] Sending process 49432 closing signal SIGTERM
W0427 14:41:01.614000 49366 site-packages/torch/distributed/elastic/multiprocessing/api.py:1012] Sending process 49433 closing signal SIGTERM
W0427 14:41:01.614000 49366 site-packages/torch/distributed/elastic/multiprocessing/api.py:1012] Sending process 49434 closing signal SIGTERM
W0427 14:41:01.615000 49366 site-packages/torch/distributed/elastic/multiprocessing/api.py:1012] Sending process 49436 closing signal SIGTERM
W0427 14:41:01.615000 49366 site-packages/torch/distributed/elastic/multiprocessing/api.py:1012] Sending process 49437 closing signal SIGTERM
W0427 14:41:01.617000 49366 site-packages/torch/distributed/elastic/multiprocessing/api.py:1012] Sending process 49438 closing signal SIGTERM
W0427 14:41:01.617000 49366 site-packages/torch/distributed/elastic/multiprocessing/api.py:1012] Sending process 49439 closing signal SIGTERM
E0427 14:41:02.873000 49366 site-packages/torch/distributed/elastic/multiprocessing/api.py:986] failed (exitcode: -6) local_rank: 3 (pid: 49435) of binary: /root/miniconda3/envs/lmf2
/bin/python3.12
```
尝试过 Qwen2.5-VL/Qwen3-VL 可以work，应当并非显存问题，显存还有很多余量
### Reproduction
启动命令
```
#!/usr/bin/env bash
set -euo pipefail
set -x
export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}"
export FORCE_TORCHRUN="${FORCE_TORCHRUN:-1}"
export NNODES="${NNODES:-1}"
export NODE_RANK="${NODE_RANK:-0}"
export NPROC_PER_NODE="${NPROC_PER_NODE:-8}"
export MASTER_ADDR="${MASTER_ADDR:-127.0.0.1}"
export MASTER_PORT="${MASTER_PORT:-29500}"
# Useful while bringing up NCCL. Override or unset in production.
export NCCL_DEBUG="${NCCL_DEBUG:-INFO}"
CONFIG="${CONFIG:-examples/train_full/qwen3_5_full_sft.yaml}"
MODEL_NAME_OR_PATH="${MODEL_NAME_OR_PATH:-Qwen/Qwen3.5-4B}"
DATASET="${DATASET:-llava_150k_en_local,llava_150k_zh_local}"
OUTPUT_DIR="${OUTPUT_DIR:-saves/qwen3.5-4b/full-zero3-single-node}"
MAX_TRAIN_SAMPLES="${MAX_TRAIN_SAMPLES:-2000}"
if [[ -z "${MAX_SAMPLES:-}" ]]; then
# LLaMA Factory applies max_samples before merging, so split the total cap across datasets.
DATASET_CSV="${DATASET//[[:space:]]/}"
DATASET_COMMAS="${DATASET_CSV//[^,]/}"
DATASET_COUNT=$((${#DATASET_COMMAS} + 1))
MAX_SAMPLES=$((MAX_TRAIN_SAMPLES / DATASET_COUNT))
if ((MAX_SAMPLES < 1)); then
MAX_SAMPLES=1
fi
fi
llamafactory-cli train "${CONFIG}" \
model_name_or_path="${MODEL_NAME_OR_PATH}" \
dataset="${DATASET}" \
max_samples="${MAX_SAMPLES}" \
output_dir="${OUTPUT_DIR}" \
finetuning_type=full \
freeze_vision_tower=false \
freeze_multi_modal_projector=false \
freeze_language_model=false \
deepspeed=examples/deepspeed/ds_z3_config.json \
flash_attn=fa2 \
per_device_train_batch_size=1 \
gradient_accumulation_steps=2 \
bf16=true \
ddp_timeout=180000000
```
```text
image_max_pixels: 262144
video_max_pixels: 16384
trust_remote_code: true
flash_attn: fa2
### method
stage: sft
do_train: true
finetuning_type: full
freeze_vision_tower: false
freeze_multi_modal_projector: false
freeze_language_model: false
deepspeed: examples/deepspeed/ds_z3_config.json
### dataset
dataset: llava_150k_en_local,llava_150k_zh_local
template: qwen3_5_nothink
cutoff_len: 2048
# max_samples is applied per dataset; 1000 x 2 datasets caps the run at 2000 samples.
max_samples: 1000
preprocessing_num_workers: 16
dataloader_num_workers: 4
### output
output_dir: saves/qwen3.5-4b/full/sft
logging_steps: 10
save_steps: 50000
plot_loss: true
overwrite_output_dir: true
save_only_model: false
report_to: none # choices: [none, wandb, tensorboard, swanlab, mlflow]
### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 2
learning_rate: 1.0e-5
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null
### eval
# val_size: 0.1
# per_device_eval_batch_size: 1
# eval_strategy: steps
# eval_steps: 500
```
### Others
_No response_
1 条评论