DataLoader multi-process initialization
bugpending
### Reminder
- [x] I have read the above rules and searched the existing issues.
### System Info
### Environment
- LLaMA-Factory version: latest (from source)
- Python version: 3.12
- Platform: Linux
### Reproduction
### Issue Description
When training with `dataloader_num_workers > 0`, the training process fails with the following error:
AttributeError: Can't get local object 'PreTrainedModel.enable_input_require_grads.<locals>.make_inputs_require_grads'
This error occurs during DataLoader multi-process initialization when pickle tries to serialize the model object.
### Root Cause Analysis
The issue originates from the `enable_input_require_grads` method in Transformers' `PreTrainedModel` class. This method registers a nested function `make_inputs_require_grads` as a forward hook:
```
def enable_input_require_grads(self):
def make_inputs_require_grads(module, input, output): # nested function (closure)
output.requires_grad_(True)
return output
self._require_grads_hook = self.register_forward_hook(make_inputs_require_grads)
```
When `dataloader_num_workers > 0`, Python's multiprocessing uses `spawn` mode which requires serializing all objects via pickle. However, **pickle cannot serialize nested functions (closures)**, leading to the AttributeError.
### Reproduction
Configuration:
```yaml
dataloader_num_workers: 4 # This triggers the error
```
Current Workaround
Setting dataloader_num_workers: 0 resolves the issue.
Suggested Solutions
1. Add documentation that dataloader_num_workers > 0 may cause this issue
2. Automatically set dataloader_num_workers: 0 when appropriate
3. Report upstream to Transformers to fix nested function serialization
### Others
_No response_
0 条评论