ITADN

DataLoader multi-process initialization

#10444Openchan-yuu 创建于 2026-04-29
bugpending
C
chan-yuucommented
### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info ### Environment - LLaMA-Factory version: latest (from source) - Python version: 3.12 - Platform: Linux ### Reproduction ### Issue Description When training with `dataloader_num_workers > 0`, the training process fails with the following error: AttributeError: Can't get local object 'PreTrainedModel.enable_input_require_grads.<locals>.make_inputs_require_grads' This error occurs during DataLoader multi-process initialization when pickle tries to serialize the model object. ### Root Cause Analysis The issue originates from the `enable_input_require_grads` method in Transformers' `PreTrainedModel` class. This method registers a nested function `make_inputs_require_grads` as a forward hook: ``` def enable_input_require_grads(self): def make_inputs_require_grads(module, input, output): # nested function (closure) output.requires_grad_(True) return output self._require_grads_hook = self.register_forward_hook(make_inputs_require_grads) ``` When `dataloader_num_workers > 0`, Python's multiprocessing uses `spawn` mode which requires serializing all objects via pickle. However, **pickle cannot serialize nested functions (closures)**, leading to the AttributeError. ### Reproduction Configuration: ```yaml dataloader_num_workers: 4 # This triggers the error ``` Current Workaround Setting dataloader_num_workers: 0 resolves the issue. Suggested Solutions 1. Add documentation that dataloader_num_workers > 0 may cause this issue 2. Automatically set dataloader_num_workers: 0 when appropriate 3. Report upstream to Transformers to fix nested function serialization ### Others _No response_
0 条评论