ITADN

Simplify TokenizerArgs __post_init__: Unnecessarily verbose

#1518ClosedJack-Khuu 创建于 2025-03-31
good first issueactionabletriaged
J
Jack-Khuucommented
### 🚀 The feature, motivation and pitch `TokenizerArgs.__post_init__` has grown quite verbose/redundant and could use a bit of simplification https://github.com/pytorch/torchchat/blob/1384f7d3d7af0847d8364fe7b300a8b49f2213c2/torchchat/cli/builder.py#L244-L289 Task: Simplify the logic in __post_init__ to reduce redundancy --- To test, run a model with each tokenizer type: - python torchchat.py generate llama2 - python torchchat.py generate llama3 - python torchchat.py generate granite-code ### Alternatives _No response_ ### Additional context _No response_ ### RFC (Optional) _No response_
关闭于 2025-04-28 9 条评论