ITADN

[BLS, VLLM and Python Backend]

#8917Closedprotonicage 创建于 13 天前
P
protonicagecommented
**Description** I have problems when using BLS to communicate with the VLLM backend. The python BLS model (which calls a VLLM model) itself is async but decoupled=False and also make an async_exec(decoupled=True) call to the VLLM model. This does not work for reasons I can not comprehend completely, but it has to do with the asyncio backend which gives me an "invalid argument" error when i try to await the requests. I found this hint in the python backend: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/bls.html#limitation For me this mean calling an decoupled python model with async_exec will not work, but this would mean your default implementation of vllm would not be usable with BLS calls. But then I found also this: https://github.com/triton-inference-server/python_backend#decoupled-mode Is this limitation of BLS still valid? **Triton Information** 26.06 NGC Container Are you using the Triton container or did you build it yourself? I use a NGC container.
关闭于 11 天前 0 条评论