Package: python3-cryptography
Version: 43.0.0-3
Severity: important
Tags: upstream
I just upgraded from Bookworm to Trixie. Our Django website, run using WSGI
under apache, no longer worked.
The apache error log showed:
from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
ImportError: PyO3 modules compiled for CPython 3.8 or older may only be
initialized once per interpreter process
I spent a long time trying to work out why this had broken. It was
quite hard to find out what the actual problem was, so I'm filing this
bug to help other people get to the bottom of it.
This issue appears to be different from the similar sounding #1078747
("python3-cryptography: cryptography.exceptions fails to import")
(which is to do with openssl-provider-legacy)
It seems that any pieces of software which uses python subinterpreters
will just fail on Trixie if they import a python module built with
PyO3, such as python3-cryptography. The list includes: mod_wsgi,
ceph, python-oracledb.
This post from March
(https://github.com/oracle/python-oracledb/issues/455#issuecomment-2691976465)
explains what is going on:
-------
The cryptography library is implemented in Rust and integrates
with Python via pyo3. pyo3 enforces strict checks to prevent
multiple instances of the same Rust-based extension from being
loaded into memory. However, mod_wsgi, by default, runs in a
subinterpreter mode, where each request may execute in a separate
Python subinterpreter. These subinterpreters have isolated memory
spaces but still attempt to load shared libraries independently.
When oracledb initializes a session pool, it triggers the loading
of cryptography again within a new subinterpreter. Since
cryptography was already loaded in a different memory space, pyo3
prevents it from being reloaded, leading to an error.
Solution
To resolve this issue, mod_wsgi should be configured to run in
main interpreter mode, preventing subinterpreters from causing
duplicate library loads. This can be done by adding the following
directive to the Apache configuration:
WSGIApplicationGroup %{GLOBAL}
This forces all requests to be executed in the main Python
interpreter, ensuring that cryptography is loaded only once and
remains accessible throughout the application lifecycle.
----
In our case I'm not sure what else on the server might be importing the
cryptography module, but there are several possibilities.
The WSGI docs say:
---------
The WSGIApplicationGroup directive can be used to specify which
application group a WSGI application or set of WSGI applications
belongs to. All WSGI applications within the same application group
will execute within the context of the same Python sub interpreter of
the process handling the request.
Any WSGI applications in the global application group will always be
executed within the context of the first interpreter created by Python
when it is initialised, of the process handling the request
---------
I'm not sure what the disadvantage of changing this setting
is. Perhaps it is less performant to just have the one interpreter
context?
It does seem quite broken that importing python3-cryptography under
WSGI (with WSGi defaults) will not work on Trixie if anything else has
also loaded it (and it seems like a lot of stuff might load it)..
The bug is really in pyo3, which simply does not yet support the
multiple interpreter contexts. https://github.com/pyca/cryptography/issues/12080
But this feels like Trixie has an incompatible set of packages/configs.
One thing I don't understand is why was this not a problem on
Bookworm, with python3-cryptography 38.0.4-3+deb12u1, which was also
implemented in rust. Did that somehow not use PyO3? Or were the WSGI
defaults different?
In our case we have switched to using python3-pyaes (a pure python
crypto implementation) instead of the rust version, in our django
application, to work around this issue, but that may not be at all
practical for many projects.
Hope this helps some people.
