FP16 model supported for CPU
When I used the builder.py to convert a local huggingface model to onnx with FP16 for CPU, I could convert it successfully.
`python -m onnxruntime_genai.models.builder -i ~/hf_models/Qwen2.5-1.5B/ -o ~/onnx_models/qwen2.5-1.5b-onnx-fp16 -e cpu -p fp16`
But when I used `model-qa.py` to run it, there is a dimension mismatch issue:
[E:onnxruntime:onnxruntime-genai, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running Cast node. Name:'InsertedPrecisionFreeCast_/model/layers.1/attn/v_proj/repeat_kv/Reshape_4/output_0' Status Message: /onnxruntime_src/onnxruntime/core/framework/op_kernel.cc:83 virtual OrtValue* onnxruntime::OpKernelContext::OutputMLValue(int, const onnxruntime::TensorShape&) status.IsOK() was false. Shape mismatch attempting to re-use buffer. {1,1,1536} != {1,25,1536}. Validate usage of dim_value (values should be > 0) and dim_param (all values with the same string should equate to the same size) in shapes in the model.
For int4 and fp32, it can work normally.
ONNXRuntime-GenAI version: v0.12.1
1 条评论