ITADN

Enable torchao.experimental EmbeddingQuantization

#1520OpenJack-Khuu 创建于 2025-03-31
Quantizationtriaged
J
Jack-Khuucommented
### 🚀 The feature, motivation and pitch Quantization is a technique used to reduce the speed, size, or memory requirements of a model and [torchao](https://github.com/pytorch/ao) is PyTorch's native quantization library for inference and training There are new experimental quantizations in torchao that we would like to enable in torchchat. Specifically this task is for enabling [EmbeddingQuantizer](https://github.com/pytorch/ao/blob/42e1345f0bc451383bcd27e39e93d4ae673eabe0/torchao/experimental/quant_api.py#L585) and [SharedEmbeddingQuantizer](https://github.com/pytorch/ao/blob/42e1345f0bc451383bcd27e39e93d4ae673eabe0/torchao/experimental/quant_api.py#L882). **Entrypoint**: https://github.com/pytorch/torchchat/blob/1384f7d3d7af0847d8364fe7b300a8b49f2213c2/torchchat/utils/quantize.py#L101 **Task**: Using ExecuTorch as a reference (https://github.com/pytorch/executorch/pull/9548) add support for EmbeddingQuantizer and SharedEmbeddingQuantizer. cc: @metascroy, @manuelcandales ### Alternatives _No response_ ### Additional context _No response_ ### RFC (Optional) _No response_
6 条评论