[Bug] v[2.8.0, 2.9.0]-cpu: Model virtual environment usage failed, 模型虚拟环境使用失败, ImportError: cannot import name 'HybridCache'...
bug
### System Info / 系統信息
windows 11 -> memory 32G
CPU -> AMD Ryzen 9 9950X 16-Core Processor
sentence-transformers -> 5.5.1
transformers -> 5.9.0
xllamacpp -> 2026.5.9093
### Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece？
- [x] docker / docker
- [ ] pip install / 通过 pip install 安装
- [ ] installation from source / 从源码安装
### Version info / 版本信息
xprobe/xinference:latest-cpu
### The command used to start Xinference / 用以启动 xinference 的命令
in dify docker-compose.yaml ->
```
services:
......
xinference:
image: xprobe/xinference:latest-cpu
container_name: xinference
ports:
- "9997:9997"
volumes:
- ./volumes/xinference:/root/.xinference
environment:
- XINFERENCE_HOME=/root/.xinference
- HF_ENDPOINT=https://hf-mirror.com
**- XINFERENCE_ENABLE_VIRTUAL_ENV=1**
- XINFERENCE_VIRTUAL_ENV_SKIP_INSTALLED=1
command: ["xinference-local", "-H", "0.0.0.0"]
restart: always
```
### Reproduction / 复现过程
### **1. cmd execute docker compose up -d -> look at the command box above**
### **2. visit localhost:9997 -> click 'Embedding Models' or 'Rerank Models' in 'Launch Model'**
### **3. choose 'bge-m3' or 'bge-reranker-v2-m3' and click launch ->**
<img width="1929" height="1312" alt="Image" src="https://github.com/user-attachments/assets/9725d8f6-ee33-4738-96b4-51d58765ad83" />
<img width="306" height="305" alt="Image" src="https://github.com/user-attachments/assets/e12c5040-2494-40f2-8916-5380a91a0094" />
<img width="804" height="651" alt="Image" src="https://github.com/user-attachments/assets/1f4de477-dbb0-411e-84b1-ffab53e96c56" />
<img width="776" height="229" alt="Image" src="https://github.com/user-attachments/assets/8ef1fa47-21b9-4a76-a7e6-c19ba381ba0b" />
### **4. view logs** ->
2026-05-28 01:57:59,875 xinference.core.model 152 [local@0.0.0.0:44148] INFO Start requests handler.
2026-05-28 01:57:59,898 xinference.core.worker 72 [local@0.0.0.0:44148] ERROR Failed to load model bge-m3-0
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/site-packages/xinference/model/embedding/sentence_transformers/core.py", line 135, in load
import sentence_transformers
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sentence_transformers/__init__.py", line 15, in <module>
from sentence_transformers.base.sampler import DefaultBatchSampler, MultiDatasetDefaultBatchSampler
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sentence_transformers/base/__init__.py", line 4, in <module>
from .model import BaseModel
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sentence_transformers/base/model.py", line 32, in <module>
from sentence_transformers.base.model_card import BaseModelCardData, generate_model_card
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sentence_transformers/base/model_card.py", line 27, in <module>
from transformers import TrainerCallback
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 2249, in __getattr__
module = self._get_module(self._class_to_module[name])
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 2483, in _get_module
raise e
File "/opt/conda/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 2481, in _get_module
return importlib.import_module("." + module_name, self.__name__)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/transformers/trainer_callback.py", line 26, in <module>
from .trainer_utils import IntervalStrategy, SaveStrategy, has_length
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/transformers/trainer_utils.py", line 66, in <module>
from peft import PeftMixedModel, PeftModel
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/peft/__init__.py", line 17, in <module>
from .auto import (
^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/peft/auto.py", line 32, in <module>
from .peft_model import (
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/peft/peft_model.py", line 37, in <module>
from transformers import Cache, DynamicCache, EncoderDecoderCache, HybridCache, PreTrainedModel
^^^^^^^^^^^^^^^^^
ImportError: cannot import name 'HybridCache' from 'transformers' (/opt/conda/lib/python3.11/site-packages/transformers/__init__.py)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/site-packages/xinference/core/worker.py", line 2123, in launch_builtin_model
await model_ref.load()
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/context.py", line 262, in send
return self._process_result_message(result)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/context.py", line 111, in _process_result_message
raise message.as_instanceof_cause()
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/pool.py", line 689, in send
result = await self._run_coro(message.message_id, coro)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/pool.py", line 389, in _run_coro
return await coro
File "/opt/conda/lib/python3.11/site-packages/xoscar/api.py", line 418, in __on_receive__
return await super().__on_receive__(message) # type: ignore
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 564, in __on_receive__
raise ex
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
async with self._lock:
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.__on_receive__
with debug_async_timeout('actor_lock_timeout',
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.__on_receive__
result = await result
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xinference/core/model.py", line 452, in load
await asyncio.to_thread(self._model.load)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xinference/model/embedding/sentence_transformers/core.py", line 150, in load
raise ImportError(f"{error_message}\n\n{''.join(installation_guide)}")
^^^^^^^^^^^^^^^^^
ImportError: [address=0.0.0.0:36025, pid=152] Failed to import module 'SentenceTransformer'
Please make sure 'sentence-transformers' is installed. You can install it by `pip install sentence-transformers`
2026-05-28 01:57:59,974 xinference.core.worker 72 [local@0.0.0.0:44148] ERROR [request 810fe4be-5a38-11f1-898e-7af218298db7] Leave launch_builtin_model, error: [address=0.0.0.0:36025, pid=152] Failed to import module 'SentenceTransformer'
Please make sure 'sentence-transformers' is installed. You can install it by `pip install sentence-transformers`
, elapsed time: 64 s
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/site-packages/xinference/model/embedding/sentence_transformers/core.py", line 135, in load
import sentence_transformers
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sentence_transformers/__init__.py", line 15, in <module>
from sentence_transformers.base.sampler import DefaultBatchSampler, MultiDatasetDefaultBatchSampler
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sentence_transformers/base/__init__.py", line 4, in <module>
from .model import BaseModel
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sentence_transformers/base/model.py", line 32, in <module>
from sentence_transformers.base.model_card import BaseModelCardData, generate_model_card
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sentence_transformers/base/model_card.py", line 27, in <module>
from transformers import TrainerCallback
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 2249, in __getattr__
module = self._get_module(self._class_to_module[name])
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 2483, in _get_module
raise e
File "/opt/conda/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 2481, in _get_module
return importlib.import_module("." + module_name, self.__name__)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/transformers/trainer_callback.py", line 26, in <module>
from .trainer_utils import IntervalStrategy, SaveStrategy, has_length
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/transformers/trainer_utils.py", line 66, in <module>
from peft import PeftMixedModel, PeftModel
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/peft/__init__.py", line 17, in <module>
from .auto import (
^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/peft/auto.py", line 32, in <module>
from .peft_model import (
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/peft/peft_model.py", line 37, in <module>
from transformers import Cache, DynamicCache, EncoderDecoderCache, HybridCache, PreTrainedModel
^^^^^^^^^^^^^^^^^
ImportError: cannot import name 'HybridCache' from 'transformers' (/opt/conda/lib/python3.11/site-packages/transformers/__init__.py)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/site-packages/xinference/core/utils.py", line 95, in wrapped
ret = await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xinference/core/worker.py", line 2123, in launch_builtin_model
await model_ref.load()
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/context.py", line 262, in send
return self._process_result_message(result)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/context.py", line 111, in _process_result_message
raise message.as_instanceof_cause()
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/pool.py", line 689, in send
result = await self._run_coro(message.message_id, coro)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/pool.py", line 389, in _run_coro
return await coro
File "/opt/conda/lib/python3.11/site-packages/xoscar/api.py", line 418, in __on_receive__
return await super().__on_receive__(message) # type: ignore
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 564, in __on_receive__
raise ex
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
async with self._lock:
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.__on_receive__
with debug_async_timeout('actor_lock_timeout',
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.__on_receive__
result = await result
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xinference/core/model.py", line 452, in load
await asyncio.to_thread(self._model.load)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xinference/model/embedding/sentence_transformers/core.py", line 150, in load
raise ImportError(f"{error_message}\n\n{''.join(installation_guide)}")
^^^^^^^^^^^^^^^^^
ImportError: [address=0.0.0.0:36025, pid=152] Failed to import module 'SentenceTransformer'
Please make sure 'sentence-transformers' is installed. You can install it by `pip install sentence-transformers`
2026-05-28 01:57:59,976 xinference.core.supervisor 72 [local@0.0.0.0:44148] ERROR Failed to launch replica bge-m3-0: [address=0.0.0.0:36025, pid=152] Failed to import module 'SentenceTransformer'
Please make sure 'sentence-transformers' is installed. You can install it by `pip install sentence-transformers`
2026-05-28 01:57:59,977 xinference.core.supervisor 72 [local@0.0.0.0:44148] ERROR [request 810fd316-5a38-11f1-898e-7af218298db7] Leave launch_builtin_model, error: [address=0.0.0.0:36025, pid=152] Failed to import module 'SentenceTransformer'
Please make sure 'sentence-transformers' is installed. You can install it by `pip install sentence-transformers`
, elapsed time: 64 s
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/site-packages/xinference/model/embedding/sentence_transformers/core.py", line 135, in load
import sentence_transformers
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sentence_transformers/__init__.py", line 15, in <module>
from sentence_transformers.base.sampler import DefaultBatchSampler, MultiDatasetDefaultBatchSampler
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sentence_transformers/base/__init__.py", line 4, in <module>
from .model import BaseModel
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sentence_transformers/base/model.py", line 32, in <module>
from sentence_transformers.base.model_card import BaseModelCardData, generate_model_card
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sentence_transformers/base/model_card.py", line 27, in <module>
from transformers import TrainerCallback
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 2249, in __getattr__
module = self._get_module(self._class_to_module[name])
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 2483, in _get_module
raise e
File "/opt/conda/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 2481, in _get_module
return importlib.import_module("." + module_name, self.__name__)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/transformers/trainer_callback.py", line 26, in <module>
from .trainer_utils import IntervalStrategy, SaveStrategy, has_length
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/transformers/trainer_utils.py", line 66, in <module>
from peft import PeftMixedModel, PeftModel
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/peft/__init__.py", line 17, in <module>
from .auto import (
^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/peft/auto.py", line 32, in <module>
from .peft_model import (
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/peft/peft_model.py", line 37, in <module>
from transformers import Cache, DynamicCache, EncoderDecoderCache, HybridCache, PreTrainedModel
^^^^^^^^^^^^^^^^^
ImportError: cannot import name 'HybridCache' from 'transformers' (/opt/conda/lib/python3.11/site-packages/transformers/__init__.py)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/site-packages/xinference/core/utils.py", line 95, in wrapped
ret = await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xinference/core/supervisor.py", line 1888, in launch_builtin_model
await _launch_model()
File "/opt/conda/lib/python3.11/site-packages/xinference/core/supervisor.py", line 1826, in _launch_model
raise result
File "/opt/conda/lib/python3.11/site-packages/xinference/core/supervisor.py", line 1625, in _launch_one_model
subpool_address = await worker_ref.launch_builtin_model(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/context.py", line 262, in send
return self._process_result_message(result)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/context.py", line 111, in _process_result_message
raise message.as_instanceof_cause()
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/pool.py", line 689, in send
result = await self._run_coro(message.message_id, coro)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/pool.py", line 389, in _run_coro
return await coro
^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/api.py", line 418, in __on_receive__
return await super().__on_receive__(message) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 564, in __on_receive__
raise ex
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
async with self._lock:
File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.__on_receive__
with debug_async_timeout('actor_lock_timeout',
File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.__on_receive__
result = await result
File "/opt/conda/lib/python3.11/site-packages/xinference/core/utils.py", line 95, in wrapped
ret = await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xinference/core/worker.py", line 2123, in launch_builtin_model
await model_ref.load()
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/context.py", line 262, in send
return self._process_result_message(result)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/context.py", line 111, in _process_result_message
raise message.as_instanceof_cause()
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/pool.py", line 689, in send
result = await self._run_coro(message.message_id, coro)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/pool.py", line 389, in _run_coro
return await coro
File "/opt/conda/lib/python3.11/site-packages/xoscar/api.py", line 418, in __on_receive__
return await super().__on_receive__(message) # type: ignore
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 564, in __on_receive__
raise ex
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
async with self._lock:
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.__on_receive__
with debug_async_timeout('actor_lock_timeout',
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.__on_receive__
result = await result
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xinference/core/model.py", line 452, in load
await asyncio.to_thread(self._model.load)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xinference/model/embedding/sentence_transformers/core.py", line 150, in load
raise ImportError(f"{error_message}\n\n{''.join(installation_guide)}")
^^^^^^^^^^^^^^^^^
ImportError: [address=0.0.0.0:36025, pid=152] Failed to import module 'SentenceTransformer'
Please make sure 'sentence-transformers' is installed. You can install it by `pip install sentence-transformers`
2026-05-28 01:57:59,980 xinference.api.restful_api 1 [local@0.0.0.0:9997] ERROR [address=0.0.0.0:36025, pid=152] Failed to import module 'SentenceTransformer'
Please make sure 'sentence-transformers' is installed. You can install it by `pip install sentence-transformers`
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/site-packages/xinference/model/embedding/sentence_transformers/core.py", line 135, in load
import sentence_transformers
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sentence_transformers/__init__.py", line 15, in <module>
from sentence_transformers.base.sampler import DefaultBatchSampler, MultiDatasetDefaultBatchSampler
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sentence_transformers/base/__init__.py", line 4, in <module>
from .model import BaseModel
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sentence_transformers/base/model.py", line 32, in <module>
from sentence_transformers.base.model_card import BaseModelCardData, generate_model_card
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sentence_transformers/base/model_card.py", line 27, in <module>
from transformers import TrainerCallback
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 2249, in __getattr__
module = self._get_module(self._class_to_module[name])
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 2483, in _get_module
raise e
File "/opt/conda/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 2481, in _get_module
return importlib.import_module("." + module_name, self.__name__)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/transformers/trainer_callback.py", line 26, in <module>
from .trainer_utils import IntervalStrategy, SaveStrategy, has_length
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/transformers/trainer_utils.py", line 66, in <module>
from peft import PeftMixedModel, PeftModel
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/peft/__init__.py", line 17, in <module>
from .auto import (
^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/peft/auto.py", line 32, in <module>
from .peft_model import (
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/peft/peft_model.py", line 37, in <module>
from transformers import Cache, DynamicCache, EncoderDecoderCache, HybridCache, PreTrainedModel
^^^^^^^^^^^^^^^^^
ImportError: cannot import name 'HybridCache' from 'transformers' (/opt/conda/lib/python3.11/site-packages/transformers/__init__.py)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/site-packages/xinference/api/restful_api.py", line 655, in launch_model
model_uid = await (await self._get_supervisor_ref()).launch_builtin_model(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/context.py", line 262, in send
return self._process_result_message(result)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/context.py", line 111, in _process_result_message
raise message.as_instanceof_cause()
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/pool.py", line 689, in send
result = await self._run_coro(message.message_id, coro)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/pool.py", line 389, in _run_coro
return await coro
File "/opt/conda/lib/python3.11/site-packages/xoscar/api.py", line 418, in __on_receive__
return await super().__on_receive__(message) # type: ignore
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 564, in __on_receive__
raise ex
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
async with self._lock:
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.__on_receive__
with debug_async_timeout('actor_lock_timeout',
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.__on_receive__
result = await result
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xinference/core/utils.py", line 95, in wrapped
ret = await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xinference/core/supervisor.py", line 1888, in launch_builtin_model
await _launch_model()
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xinference/core/supervisor.py", line 1826, in _launch_model
raise result
File "/opt/conda/lib/python3.11/site-packages/xinference/core/supervisor.py", line 1625, in _launch_one_model
subpool_address = await worker_ref.launch_builtin_model(
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/context.py", line 262, in send
return self._process_result_message(result)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/context.py", line 111, in _process_result_message
raise message.as_instanceof_cause()
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/pool.py", line 689, in send
result = await self._run_coro(message.message_id, coro)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/pool.py", line 389, in _run_coro
return await coro
File "/opt/conda/lib/python3.11/site-packages/xoscar/api.py", line 418, in __on_receive__
return await super().__on_receive__(message) # type: ignore
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 564, in __on_receive__
raise ex
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
async with self._lock:
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.__on_receive__
with debug_async_timeout('actor_lock_timeout',
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.__on_receive__
result = await result
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xinference/core/utils.py", line 95, in wrapped
ret = await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xinference/core/worker.py", line 2123, in launch_builtin_model
await model_ref.load()
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/context.py", line 262, in send
return self._process_result_message(result)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/context.py", line 111, in _process_result_message
raise message.as_instanceof_cause()
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/pool.py", line 689, in send
result = await self._run_coro(message.message_id, coro)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xoscar/backends/pool.py", line 389, in _run_coro
return await coro
File "/opt/conda/lib/python3.11/site-packages/xoscar/api.py", line 418, in __on_receive__
return await super().__on_receive__(message) # type: ignore
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 564, in __on_receive__
raise ex
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
async with self._lock:
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.__on_receive__
with debug_async_timeout('actor_lock_timeout',
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.__on_receive__
result = await result
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xinference/core/model.py", line 452, in load
await asyncio.to_thread(self._model.load)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/xinference/model/embedding/sentence_transformers/core.py", line 150, in load
raise ImportError(f"{error_message}\n\n{''.join(installation_guide)}")
^^^^^^^^^^^^^^^^^
ImportError: [address=0.0.0.0:36025, pid=152] Failed to import module 'SentenceTransformer'
Please make sure 'sentence-transformers' is installed. You can install it by `pip install sentence-transformers`
### Expected behavior / 期待表现
Why didn't use a model of own environment?
My startup parameters include: - XINFERENCE_ENABLE_VIRTUAL_ENV=1
I even made a separate selection on the model page: Model Virtual Environment -> true or unset
But look the logs, although he has a 3.11.5 directory in directory “.xinference\virtualenv\v4\bge-m3\sentence_transformers”, but it seems that he is still invoke the global environment(File "/opt/conda/lib/python3.11/site-packages/xinference/model......")?
But the LLM model, such as qwen2.5, is good
1 条评论