ITADN

Improve Tokenizer New Type Onboarding

#1536Openzhenyan-zhang-meta 创建于 2025-04-28
good first issueactionabletriaged
### 🚀 The feature, motivation and pitch --- As a sequel to https://github.com/pytorch/torchchat/issues/1518 where we added an enum for tokenizer types to simplify `TokenizerArgs __post_init__`, we need to further improve it to simplify new tokenizer type onboarding: ### Tasks --- - Move TokenizerType to a centralized place - We now have two of them: https://github.com/pytorch/torchchat/blob/0299a37a342348803763e37e9f4823c5bcb12c92/dist_run.py#L67-L69 https://github.com/pytorch/torchchat/blob/0299a37a342348803763e37e9f4823c5bcb12c92/torchchat/cli/builder.py#L241-L245 - Check all getters of tokenizer types - It may be able to be simplified as inline https://github.com/pytorch/torchchat/blob/0299a37a342348803763e37e9f4823c5bcb12c92/torchchat/generate.py#L369 - Add documentation for future tokenizer onboard. - We may need to point people to update the model validation logic: https://github.com/pytorch/torchchat/blob/0299a37a342348803763e37e9f4823c5bcb12c92/torchchat/cli/builder.py#L290-L322 --- To test, run a model with each tokenizer type: - python torchchat.py generate llama2 - python torchchat.py generate llama3 - python torchchat.py generate granite-code cc @Jack-Khuu @byjlw
3 条评论