在winserver2022中遇到了问题

# 我在训练时接连遇到了三个问题 ## 训练环境： - 系统winserver2022 - 显卡数量2 - 显卡型号Tesa P40 24G *2 - conda环境配置，一直按照https://ecn4x4y9jhjs.feishu.cn/wiki/J9RWwBIULigtX7k7PZMcWQV7nzd走，同样，数据集制作也是同一个实例、环境、系统 - 配置文件： ``` base_config: - configs/acoustic.yaml dictionaries: zh: dictionaries/opencpop-extension.txt extra_phonemes: [] merged_phoneme_groups: [] datasets: - raw_data_dir: data/funingna/raw speaker: funingna spk_id: 0 language: zh test_prefixes: - 0_024 - 0_058 - 0_151 - 0_040 - 0_087 - 0_195 - 0_052 - 0_129 - 0_280 - 0_219 binary_data_dir: data/aco_funingna/binary binarization_args: num_workers: 0 pe: rmvpe pe_ckpt: 'checkpoints/rmvpe/model.pt' hnsep: vr hnsep_ckpt: 'checkpoints/vr/model.pt' vocoder: NsfHifiGAN vocoder_ckpt: checkpoints/nsf_hifigan_44.1k_hop512_128bin_2024.02/model.ckpt use_lang_id: false num_lang: 1 use_spk_id: false num_spk: 1 # NOTICE: before enabling variance embeddings, please read the docs at # https://github.com/openvpi/DiffSinger/tree/main/docs/BestPractices.md#choosing-variance-parameters use_energy_embed: false use_breathiness_embed: true use_voicing_embed: true use_tension_embed: true use_key_shift_embed: true use_speed_embed: true augmentation_args: random_pitch_shifting: enabled: true range: [-5., 5.] scale: 0.75 fixed_pitch_shifting: enabled: false targets: [-5., 5.] scale: 0.5 random_time_stretching: enabled: true range: [0.5, 2.] scale: 0.75 # diffusion and shallow diffusion diffusion_type: reflow enc_ffn_kernel_size: 3 use_rope: true use_shallow_diffusion: true T_start: 0.4 T_start_infer: 0.4 K_step: 300 K_step_infer: 300 backbone_type: 'lynxnet' backbone_args: num_channels: 1024 num_layers: 6 kernel_size: 31 dropout_rate: 0.0 strong_cond: true #backbone_type: 'wavenet' #backbone_args: # num_channels: 512 # num_layers: 20 # dilation_cycle_length: 4 shallow_diffusion_args: train_aux_decoder: true train_diffusion: true val_gt_start: false aux_decoder_arch: convnext aux_decoder_args: num_channels: 512 num_layers: 6 kernel_size: 7 dropout_rate: 0.1 aux_decoder_grad: 0.1 lambda_aux_mel_loss: 0.2 optimizer_args: lr: 0.0006 lr_scheduler_args: scheduler_cls: torch.optim.lr_scheduler.StepLR step_size: 10000 gamma: 0.75 max_batch_frames: 50000 max_batch_size: 64 max_updates: 320000 num_valid_plots: 10 val_with_vocoder: true val_check_interval: 2000 num_ckpt_keep: 5 permanent_ckpt_start: 120000 permanent_ckpt_interval: 20000 pl_trainer_devices: 'auto' pl_trainer_precision: '16-mixed' ``` ## 问题1： RuntimeError: Distributed package doesn't have NCCL built in ### 我的解决方案： ``` # ---- 强制 monkey patch torch.distributed.init_process_group ---- import torch.distributed _original_init_process_group = torch.distributed.init_process_group def _patched_init_process_group(backend='nccl', *args, **kwargs): print("Forcing distributed backend to 'gloo' (was: {})".format(backend)) return _original_init_process_group(backend='gloo', *args, **kwargs) torch.distributed.init_process_group = _patched_init_process_group # --------------------------------------------------------------- ``` ## 问题2： ``` Traceback (most recent call last): File "C:\ProgramData\miniconda3\envs\diffsinger\lib\multiprocessing\queues.py", line 244, in _feed obj = _ForkingPickler.dumps(obj) File "C:\ProgramData\miniconda3\envs\diffsinger\lib\multiprocessing\reduction.py", line 51, in dumps cls(buf, protocol).dump(obj) File "C:\ProgramData\miniconda3\envs\diffsinger\lib\site-packages\torch\multiprocessing\reductions.py", line 607, in reduce_storage metadata = storage._share_filename_cpu_() File "C:\ProgramData\miniconda3\envs\diffsinger\lib\site-packages\torch\storage.py", line 447, in wrapper return fn(self, *args, **kwargs) File "C:\ProgramData\miniconda3\envs\diffsinger\lib\site-packages\torch\storage.py", line 526, in _share_filename_cpu_ return super()._share_filename_cpu_(*args, **kwargs) RuntimeError: Couldn't open shared file mapping: <torch_788_162147526_13>, error code: <1455> ``` ### 我的解决方案： ``` if __name__ == '__main__': import torch.multiprocessing as mp mp.set_start_method('spawn') run_task() ``` ## 问题3： ``` 296.059 Total estimated model params size (MB) Epoch 0: 0%| | 0/21 [00:00<?, ?it/s][rank: 1] Child process with PID 8108 terminated with code 3221225477. Forcefully terminating all other processes to avoid zombies 🧟 ``` ### 解决方案： 最终我折腾了1个多小时 退还实例，换成Ubuntu，结果未知