ITADN

Adding support for subinterpreters

#2159Openmwaddoups 创建于 2026-02-25
M
mwaddoupscommented
Currently, pyzmq does not support subinterpreters (from `concurrent.subinterpreters`) - these were introduced in Python 3.12 and made easier to access in Python 3.14. These allow for full context isolation, and since the interpreters don't share a GIL they can use separate cores when threaded without being in free-threaded mode. The import just fails as the C extensions have not been marked as supporting subinterpreters. An example simple test script below. ```python from concurrent import interpreters from contextlib import closing def echo_responder(ctx_addr: int): import zmq shadowd_ctx = zmq.Context.shadow(ctx_addr) with shadowd_ctx.socket(zmq.DEALER) as rep_sock: rep_sock.bind("inproc://test") rep_sock.setsockopt(zmq.RCVTIMEO, 500) while input := rep_sock.recv(): rep_sock.send(input) import zmq with closing(interpreters.create()) as responder: with zmq.Context() as ctx: sock = ctx.socket(zmq.DEALER) sock.connect("inproc://test") t = responder.call_in_thread(echo_responder, ctx.underlying) for _ in range(100): sock.send(b"DATA") sock.recv() sock.send(b"") t.join() ``` This raises ``` ImportError: module zmq.backend.cython._zmq does not support loading in subinterpreters ``` However, I have been able to recompile ZMQ and get thing to work. The requirements for module isolation are noted [here](https://docs.python.org/3/howto/isolating-extensions.html) , but Cython has experimental support for enabling module state by CFLAGS. I made the following changes - Add `# subinterpreters_compatible: own_gil` to the zmq Cython backend header. - Add `CYTHON_USE_MODULE_STATE=1` as a CFLAG. This is noted as experimental, but mainly that it doesn't move global cdefs into state just yet. - Add `CYTHON_USE_TYPE_SPECS=1` as a CFLAG. This I think is needed because of the use of `cdef class`, and without this the module imports but errors on import. However, with these 3 changes the above code works and is able to share a context. I'm happy to submit a PR for the above - but I'm not sure what's preferred from the project. Since some of this seems to be experimental on cython side, would it be better to have as an opt-in feature (e.g. `PYZMQ_CYTHON_SUBINTERPRETERS`)? I'm also not an expert on these matters, so there may be more to this! Thanks for your work on a great project.
1 条评论