On Tue, Feb 21, 2023 at 02:23:34PM +0200, Adrian Bunk wrote:
> Looking at #1028371, should generated dependencies on python3-protobuf be
> python3-protobuf (>= 3.21), python3-protobuf (<< 3.22)
> to ensure that the binary package is used with the same version
> as the protobuf-compiler used during the build?
I'm not the maintainer, but a drive-by contributor. I looked a bit into
this, given its RC severity.
With my still somewhat limited understanding, a strict version alignment
between protobuf-compiler and python3-protobuf would probably resolve
this particular symptom, but the issues here seem to run deeper.
Specifically:
* The protobuf project provides three different versions of Python
bindings: pure Python, C++, and libupb-based[1]. These are
selectable using the PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION
environment variable.
* Debian's python3-protobuf, from src:protobuf, ships the pure Python
version, as well as the C++ bindings. The default implementation in
Debian is "cpp".
* The upb implementation is not included in src:protobuf, but in the
upb upstream source[2], i.e. what is src:upb in Debian, even though
the snapshot we have in Debian does not contain sources to Python
bindings.
* Upstream has switched the default implementation to "upb", and
deprecated the "cpp" implementation. There is, in fact, no way for
one to fetch the "cpp" version from PyPI. This is documented
extensively in their May 2022 release notes[3]. However, Debian
still ships, and defaults to, cpp, a major departure from upstream.
* Relatedly, when they made that switch, they also made changes to
their versioning scheme, disconnecting the Python library's version
from the source version. As a result, the Python API (both upb, as
well as pure Python), is now versioned at "4.21", rather than
"3.21". The Debian binary package python3-protobuf is versioned as
"3.21.12-1", which is not a version that exists, or will ever exist,
upstream. That binary package in fact, is shipping an egg named
protobuf-4.21.12.egg-info. (This is all also well documented in their
release notes[3]).
* Finally, in the same release notes document[3], they also state:
"Python upb requires generated code that has been generated from
protoc 3.19.0 or newer.".
Indeed, if one fetches protobuf 4.21 from PyPI, and runs:
python3 -c 'import bernhard'
or
PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=upb python3 -c 'import bernhard'
...a traceback message is emitted, but a much more informative one:
> TypeError: Descriptors cannot not be created directly.
>
> If this call came from a _pb2.py file, your generated code is out of
> date and must be regenerated with protoc >= 3.19.0.
>
> If you cannot immediately regenerate your protos, some other possible
workarounds are:
> 1. Downgrade the protobuf package to 3.20.x or lower.
> 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will
> use pure-Python parsing and will be much slower).
>
> More information:
https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
* The release notes specifically mention "upb" requiring protoc
(protobuf-compiler) >= 3.19, but not "cpp". However, as established
above, "cpp" is deprecated and not used by anyone (but Debian), and
therefore they either meant "the non-Pure-Python implementation"
there, or did not pay as much attention to forward- and
backwards-compatibility, or informative error messages for their
deprecated backend. It's likely, but not entirely clear, that the
protoc dependency requirement is >= 3.19 here as well.
* Finally note that the 3.21.12-1+b2 Python implementation still works
with python3-bernhard, Built-Using: protobuf-compiler (= 3.12.4-1+b3):
PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python python3 -c 'import bernhard'
All in all: it's almost certainly necessary to make the dependency
tighter, to something like >= 3.19, if not tight to = 3.21.
I still feel uneasy about Debian shipping a version of python3-protobuf
that includes, and defaults to, an implementation that is deprecated
upstream (and on top of it, is misversioned). I'm not sure what to make
of this so late in the release cycle, though.
For trixie the path forward is probably something along the lines of
updating src:upb to a newer upstream, building the upb-based extension
as python3-protobuf-upb, and then changing src:protobuf to not build the
cpp extension, make python3-protobuf Arch: all, and then Recommend (or
Depend) on python3-protobuf-upb as the native/fast implementation.
Faidon
1: https://github.com/protocolbuffers/protobuf/tree/main/python
2: https://github.com/protocolbuffers/upb/tree/main/python
3: https://protobuf.dev/news/2022-05-06/
