Simplify TokenizerArgs __post_init__: Unnecessarily verbose
good first issueactionabletriaged
### 🚀 The feature, motivation and pitch
`TokenizerArgs.__post_init__` has grown quite verbose/redundant and could use a bit of simplification
https://github.com/pytorch/torchchat/blob/1384f7d3d7af0847d8364fe7b300a8b49f2213c2/torchchat/cli/builder.py#L244-L289
Task: Simplify the logic in __post_init__ to reduce redundancy
---
To test, run a model with each tokenizer type:
- python torchchat.py generate llama2
- python torchchat.py generate llama3
- python torchchat.py generate granite-code
### Alternatives
_No response_
### Additional context
_No response_
### RFC (Optional)
_No response_
关闭于 2025-04-28 9 条评论