Error when using LlamaCppServerProvider
Hello,
I have trouble when I start using this llama-cpp-agent
I run serve using followed command
```bash
python3 -m llama_cpp.server --model /app/vlm_weights/Qwen2-0___5B-Instruct-GGUF/qwen2-0_5b-instruct-fp16.gguf --n_gpu_layers -1
```
then I can use followed code the request
```python
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="sk-xxx")
response = client.chat.completions.create(
model="qwen2",
messages=[
{
"role": "user",
"content": "hello" # 这是文本内容
}
],
)
print(response)
```
with excepted output
```bash
root@ubuntu:/app/inside_container/llama_python_demo# python3 demo.py
ChatCompletion(id='chatcmpl-64ea1c64-a6fb-46b2-ba36-8942e9a17540', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='Hello! How can I assist you today?', refusal=None, role='assistant', function_call=None, tool_calls=None))], created=1726667242, model='qwen2', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=9, prompt_tokens=20, total_tokens=29, completion_tokens_details=None))
```
then when I used following code try to start using llama-cpp-agent
```python
from llama_cpp_agent import LlamaCppAgent
from llama_cpp_agent import MessagesFormatterType
from llama_cpp_agent.providers import LlamaCppServerProvider
provider = LlamaCppServerProvider("http://127.0.0.1:8000", llama_cpp_python_server=True)
agent = LlamaCppAgent(
provider,
system_prompt="You are a helpful assistant.",
predefined_messages_formatter_type=MessagesFormatterType.CHATML,
)
settings = provider.get_provider_default_settings()
settings.n_predict = 512
settings.temperature = 0.65
while True:
user_input = input(">")
if user_input == "exit":
break
agent_output = agent.get_chat_response(user_input, llm_sampling_settings=settings)
print(f"Agent: {agent_output.strip()}")
```
I see the error
```
root@ubuntu:/app/inside_container/llama_python_demo# python3 demo.py
>hello
Traceback (most recent call last):
File "/app/inside_container/llama_python_demo/demo.py", line 21, in <module>
agent_output = agent.get_chat_response(user_input, llm_sampling_settings=settings)
File "/usr/local/lib/python3.10/dist-packages/llama_cpp_agent/llm_agent.py", line 334, in get_chat_response
for out in completion:
File "/usr/local/lib/python3.10/dist-packages/llama_cpp_agent/providers/llama_cpp_server.py", line 279, in generate_text_chunks
new_data = json.loads(decoded_chunk.replace("data:", ""))
File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 3 (char 2)
```
Could you help me look at it?
Best wishes
0 条评论