Improve Tokenizer New Type Onboarding
good first issueactionabletriaged
### 🚀 The feature, motivation and pitch
---
As a sequel to https://github.com/pytorch/torchchat/issues/1518 where we added an enum for tokenizer types to simplify `TokenizerArgs __post_init__`, we need to further improve it to simplify new tokenizer type onboarding:
### Tasks
---
- Move TokenizerType to a centralized place
- We now have two of them: https://github.com/pytorch/torchchat/blob/0299a37a342348803763e37e9f4823c5bcb12c92/dist_run.py#L67-L69 https://github.com/pytorch/torchchat/blob/0299a37a342348803763e37e9f4823c5bcb12c92/torchchat/cli/builder.py#L241-L245
- Check all getters of tokenizer types
- It may be able to be simplified as inline https://github.com/pytorch/torchchat/blob/0299a37a342348803763e37e9f4823c5bcb12c92/torchchat/generate.py#L369
- Add documentation for future tokenizer onboard.
- We may need to point people to update the model validation logic: https://github.com/pytorch/torchchat/blob/0299a37a342348803763e37e9f4823c5bcb12c92/torchchat/cli/builder.py#L290-L322
---
To test, run a model with each tokenizer type:
- python torchchat.py generate llama2
- python torchchat.py generate llama3
- python torchchat.py generate granite-code
cc @Jack-Khuu @byjlw
3 条评论