阿里云盘中的om文件未能初始化获取到输出
Ascend310B1,CANN7.0
请问为什么在阿里云盘中使用的tiny-llama在推理时没能够正确读取模型并获得输出呢?
> (inference) root@orangepiaipro-20t:~/Desktop/lab3/ascend-llm/inference# python main.py --model /root/Desktop/tinyLlama-1.1B-Chat/tiny-llama-seq-1-key-1024-int8.om --h
f-dir /root/Desktop/tinyLlama-1.1B-Chat --engine acl --sampling greedy --cli
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
InferenceConfig(tokenizer='/root/Desktop/tinyLlama-1.1B-Chat', hf_model_dir='/root/Desktop/tinyLlama-1.1B-Chat', sampling_method='greedy', sampling_value=10, temperature=0.7, max_length=512, max_input_len=1, session_type='acl', device=0, prompt=[{'role': 'user', 'content': 'Hey there I am a human that would like to have a conversation with you.'}, {'role': 'assistant', 'content': 'Sure, I am happy to answer your questions'}, {'role': 'user', 'content': 'Great, I insist that we take turns.'}, {'role': 'assistant', 'content': 'I agree, we should take turns.'}], model='/root/Desktop/tinyLlama-1.1B-Chat/tiny-llama-seq-1-key-1024-int8.om', kvcache_method='sliding-window', kvcache_fixsize=True, head_len=32, recent_len=32, evict_len=64, n_layer=22, format='huggingface-tensor', max_cache_size=256, head_num=4, num_kv_group=8, head_dim=64, hidden_dim=2048, dtype='float16', model_type='tiny-llama')
return code is 500002, detail: load model
return code is 545008, detail: get model desc
[]
Traceback (most recent call last):
File "/root/Desktop/lab3/ascend-llm/inference/main.py", line 94, in <module>
engine = LlamaInterface(cfg)
File "/root/Desktop/lab3/ascend-llm/inference/inference.py", line 17, in __init__
self.session=Session.fromConfig(config)
File "/root/Desktop/lab3/ascend-llm/inference/session.py", line 20, in fromConfig
return AclSession(config)
File "/root/Desktop/lab3/ascend-llm/inference/session.py", line 76, in __init__
self.input_ids,_,_,self.kvCache.kvCache = self.model.getInputs()
ValueError: not enough values to unpack (expected 4, got 0)
ls /root/Desktop/tinyLlama-1.1B-Chat
README.md configuration.json gitattributes special_tokens_map.json tokenizer.json tokenizer_config.json
config.json generation_config.json quantize_config.json tiny-llama-seq-1-key-1024-int8.om tokenizer.model
0 条评论