OptiSpeech: 轻量级端到端文本转语音模型

OptiSpeech 旨在成为一个高效、轻量级且快速的端侧文本转语音模型。

我要感谢 Pneuma Solutions 为训练此模型提供 GPU 资源。他们的支持显著加速了我的开发过程。

音频示例

https://github.com/user-attachments/assets/e5001404-100f-4453-b979-8ea7d4b44659

https://github.com/user-attachments/assets/7a0d7ff8-a02c-4e8a-a38f-3b083c7c28d4

请注意，这仍然是 WIP。最终模型设计决策仍在制定中。

安装

仅推理

如果你想要一个仅用于推理的最小依赖包，且不需要 pytorch，你可以使用 ospeech

训练和开发

我们使用 uv 来管理 python 运行时和依赖项。

首先安装 uv，然后运行以下命令：

$ git clone https://github.com/mush42/optispeech
$ cd optispeech
$ uv sync

推理

命令行 API

$ python3 -m optispeech.infer  --help
usage: infer.py [-h] [--d-factor D_FACTOR] [--p-factor P_FACTOR] [--e-factor E_FACTOR] [--cuda]
                checkpoint text output_dir

Speaking text using OptiSpeech

positional arguments:
  checkpoint           Path to OptiSpeech checkpoint
  text                 Text to synthesise
  output_dir           Directory to write generated audio to.

options:
  -h, --help           show this help message and exit
  --d-factor D_FACTOR  Scale to control speech rate
  --p-factor P_FACTOR  Scale to control pitch
  --e-factor E_FACTOR  Scale to control energy
  --cuda               Use GPU for inference

Python API

import soundfile as sf
from optispeech.model import OptiSpeech

# Load model
device = torch.device("cpu")
ckpt_path = "/path/to/checkpoint"
model = OptiSpeech.load_from_checkpoint(ckpt_path, map_location="cpu")
model = model.to(device)
model = model.eval()

# Text preprocessing and phonemization
sentence = "A rainbow is a meteorological phenomenon that is caused by reflection, refraction and dispersion of light in water droplets resulting in a spectrum of light appearing in the sky."
inference_inputs = model.prepare_input(sentence)
inference_outputs = model.synthesize(inference_inputs)

inference_outputs = inference_outputs.as_numpy()
wav = inference_outputs.wav
sf.write("output.wav", wav.squeeze(), model.sample_rate)

训练

由于此代码使用了 Lightning-Hydra-Template，你便拥有了它所带来的所有功能。

训练只需 1、2、3 三步：

1. 准备数据集

假设数据集的组织方式如下：

├── train
│   ├── metadata.csv
│   └── wav
│       ├── aud-00001-0003.wav
│       └── ...
└── val
    ├── metadata.csv
    └── wav
        ├── aud-00764.wav
        └── ...

metadata.csv 文件可以包含 2、3 或 4 列，列之间以 |（竖线字符）分隔，格式如下：

2 列：file_id|text
3 列：file_id|speaker_id|text
4 列：file_id|speaker_id|language_id|text

使用 preprocess_dataset 脚本为训练准备数据集：

$ python3 -m optispeech.tools.preprocess_dataset --help
usage: preprocess_dataset.py [-h] [--format {ljspeech}] dataset input_dir output_dir

positional arguments:
  dataset              dataset config relative to `configs/data/` (without the suffix)
  input_dir            original data directory
  output_dir           Output directory to write datafiles + train.txt and val.txt

options:
  -h, --help           show this help message and exit
  --format {ljspeech}  Dataset format.

如果您正在使用新数据集进行训练，您必须使用以下脚本计算并添加 **data_statistics **：

$ python3 -m optispeech.tools.generate_data_statistics --help
usage: generate_data_statistics.py [-h] [-b BATCH_SIZE] [-f] [-o OUTPUT_DIR] input_config

positional arguments:
  input_config          The name of the yaml config file under configs/data

options:
  -h, --help            show this help message and exit
  -b BATCH_SIZE, --batch-size BATCH_SIZE
                        Can have increased batch size for faster computation
  -f, --force           force overwrite the file
  -o OUTPUT_DIR, --output-dir OUTPUT_DIR
                        Output directory to save the data statistics

2. [Optional] Choose your backbone

OptiSpeech 为模型的 encoder 和 decoder 提供可互换的 backbone 类型，您可以根据目标性能配置选择 backbone。

为了帮助您选择，以下是可用 backbone 的计算复杂度快速分析：

Backbone	Config File	FLOPs	MACs	#Params
ConvNeXt	`optispeech.yaml`	10.57 GFLOPS	5.27 GMACs	15.89 M
Light	`light.yaml`	7.88 GFLOPS	3.93 GMACs	10.74 M
Transformer	`transformer.yaml`	14.15 GFLOPS	7.06 GMACs	17.98 M
Conformer	`conformer.yaml`	20.42 GFLOPS	10.19 GMACs	24.35 M

默认 backbone 是 ConvNeXt，但如果您想更改它，可以编辑您的实验配置。

3. Start training

要开始训练，请运行以下命令。请注意，此训练运行使用了来自 hfc_female-en_US 的 config。您可以复制并使用自己的配置值进行更新，然后传递自定义配置文件（不含扩展名）的名称。

$ python3 -m optispeech.train experiment=hfc_female-en_us

ONNX 支持

ONNX 导出

$ python3 -m optispeech.onnx.export --help
usage: export.py [-h] [--opset OPSET] [--seed SEED] checkpoint_path output

Export OptiSpeech checkpoints to ONNX

positional arguments:
  checkpoint_path  Path to the model checkpoint
  output           Path to output `.onnx` file

options:
  -h, --help       show this help message and exit
  --opset OPSET    ONNX opset version to use (default 15
  --seed SEED      Random seed

ONNX 推理

$ python3 -m optispeech.onnx.infer --help
usage: infer.py [-h] [--d-factor D_FACTOR] [--p-factor P_FACTOR] [--e-factor E_FACTOR] [--cuda]
                onnx_path text output_dir

ONNX inference of OptiSpeech

positional arguments:
  onnx_path            Path to the exported LeanSpeech ONNX model
  text                 Text to speak
  output_dir           Directory to write generated audio to.

options:
  -h, --help           show this help message and exit
  --d-factor D_FACTOR  Scale to control speech rate.
  --p-factor P_FACTOR  Scale to control pitch.
  --e-factor E_FACTOR  Scale to control energy.
  --cuda               Use GPU for inference

致谢

我想要致谢的仓库：

BetterFastspeech2: 作为仓库主干
LightSpeech: 作为 transformer 主干
JETS: 作为音素-梅尔频谱对齐框架
Vocos: 在 TTS 中率先使用 ConvNext
Piper-TTS: 在端侧 TTS 领域引领潮流。同时也感谢其出色的 phonemizer

参考文献

@inproceedings{luo2021lightspeech,
    title={Lightspeech: Lightweight and fast text to speech with neural architecture search},
    author={Luo, Renqian and Tan, Xu and Wang, Rui and Qin, Tao and Li, Jinzhu and Zhao, Sheng and Chen, Enhong and Liu, Tie-Yan},
    booktitle={ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
    pages={5699--5703},
    year={2021},
    organization={IEEE}
}

@article{siuzdak2023vocos,
  title={Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis},
  author={Siuzdak, Hubert},
  journal={arXiv preprint arXiv:2306.00814},
  year={2023}
}

@INPROCEEDINGS{10446890,
  author={Okamoto, Takuma and Ohtani, Yamato and Toda, Tomoki and Kawai, Hisashi},
  booktitle={ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  title={Convnext-TTS And Convnext-VC: Convnext-Based Fast End-To-End Sequence-To-Sequence Text-To-Speech And Voice Conversion},
  year={2024},
  volume={},
  number={},
  pages={12456-12460},
  keywords={Vocoders;Neural networks;Signal processing;Transformers;Real-time systems;Acoustics;Decoding;ConvNeXt;JETS;text-to-speech;voice conversion;WaveNeXt},
  doi={10.1109/ICASSP48485.2024.10446890}
}