ITADN

FP16 model supported for CPU

#2009Openbohuazou 创建于 2026-03-05
B
bohuazoucommented
When I used the builder.py to convert a local huggingface model to onnx with FP16 for CPU, I could convert it successfully. `python -m onnxruntime_genai.models.builder -i ~/hf_models/Qwen2.5-1.5B/ -o ~/onnx_models/qwen2.5-1.5b-onnx-fp16 -e cpu -p fp16` But when I used `model-qa.py` to run it, there is a dimension mismatch issue: [E:onnxruntime:onnxruntime-genai, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running Cast node. Name:'InsertedPrecisionFreeCast_/model/layers.1/attn/v_proj/repeat_kv/Reshape_4/output_0' Status Message: /onnxruntime_src/onnxruntime/core/framework/op_kernel.cc:83 virtual OrtValue* onnxruntime::OpKernelContext::OutputMLValue(int, const onnxruntime::TensorShape&) status.IsOK() was false. Shape mismatch attempting to re-use buffer. {1,1,1536} != {1,25,1536}. Validate usage of dim_value (values should be > 0) and dim_param (all values with the same string should equate to the same size) in shapes in the model. For int4 and fp32, it can work normally. ONNXRuntime-GenAI version: v0.12.1
1 条评论