Taking over DPMT (Was: python-boto: autopkgtest failure with Python 3.8 as default)
Hi Emmanuel,
thanks a lot for your patch. Please note that you crafted it against
GIt which is lagging behind the latest NMU. I intended to apply it
anyway - but it does not solve
==
ERROR: test_constructor (tests.unit.utils.test_utils.TestPassword)
--
Traceback (most recent call last):
File
"/tmp/autopkgtest.Trbllh/autopkgtest_tmp/tests/unit/utils/test_utils.py", line
104, in test_constructor
password.set('foo')
File "/usr/lib/python3/dist-packages/boto/utils.py", line 787, in set
self.str = self.hashfunc(value).hexdigest()
File
"/tmp/autopkgtest.Trbllh/autopkgtest_tmp/tests/unit/utils/test_utils.py", line
101, in
hmac_hashfunc = lambda msg: hmac.new(b'mysecretkey', msg)
File "/usr/lib/python3.8/hmac.py", line 153, in new
return HMAC(key, msg, digestmod)
File "/usr/lib/python3.8/hmac.py", line 51, in __init__
raise TypeError("Missing required parameter 'digestmod'.")
TypeError: Missing required parameter 'digestmod'.
==
ERROR: test_hmac (tests.unit.utils.test_utils.TestPassword)
--
Traceback (most recent call last):
File
"/tmp/autopkgtest.Trbllh/autopkgtest_tmp/tests/unit/utils/test_utils.py", line
93, in test_hmac
self.clstest(HMACPassword)
File
"/tmp/autopkgtest.Trbllh/autopkgtest_tmp/tests/unit/utils/test_utils.py", line
61, in clstest
self.assertNotEquals(password, 'foo')
File "/usr/lib/python3.8/unittest/case.py", line 1410, in deprecated_func
return original_func(*args, **kwargs)
File "/usr/lib/python3.8/unittest/case.py", line 918, in assertNotEqual
if not first != second:
File "/usr/lib/python3/dist-packages/boto/utils.py", line 797, in __eq__
return str(self.hashfunc(other).hexdigest()) == str(self.str)
File
"/tmp/autopkgtest.Trbllh/autopkgtest_tmp/tests/unit/utils/test_utils.py", line
88, in hmac_hashfunc
return hmac.new(b'mysecretkey', msg)
File "/usr/lib/python3.8/hmac.py", line 153, in new
return HMAC(key, msg, digestmod)
File "/usr/lib/python3.8/hmac.py", line 51, in __init__
raise TypeError("Missing required parameter 'digestmod'.")
TypeError: Missing required parameter 'digestmod'.
--
Ran 1730 tests in 5.640s
FAILED (SKIP=6, errors=3)
On Tue, Mar 24, 2020 at 11:10:28AM -0300, eamanu wrote:
> I prepare a non-maintainer upload patch that I attach.
>
> Please consider apply it.
>
> Also I make a MR [0]
>
> [0] https://salsa.debian.org/eevans/python-boto/-/merge_requests/1
I wonder whether we should take over python-boto into DPMT maintenance
which would enable commits to Git way more easily.
Kind regards
Andreas.
--
http://fam-tille.de
Python3 modules not built for all supported Python versions
Hi, We've just finished the transition to python3.8 as the default python3 interpreter, which was a bit difficult due to some autopkgtest regressions in a few rdeps, and to the fact that many modules only build their extensions for the default python version, which means they have a strict dependency on the python3 version[1] and they need to be rebuilt and migrated in lockstep with python3-defaults. I have analyzed this based on current sid amd64 contents and have come up with the following packages that don't ship extensions for both py3.7 and 3.8 (which are the currently supported versions). Note that pure python packages that don't build C extensions are not affected. It would be great if this situation can be improved in order to help with future python transitions. Building for all the supported python versions can be done by build-depending on python3-all-dev and compiling your package (or just the python bits) with PYTHON pointing to each version. Depending on your package's build system, this could be largely automated using some helper, such as pybuild. If you don't know how to add support for your package, feel free to ask. Cheers, Emilio [1] e.g. python3 (>= 3.7), python3 (<< 3.8) "Adam C. Powell, IV" netgen (U) A. Maitland Bottoms gr-air-modes gr-fcdproplus (U) gr-fosphor gr-gsm (U) gr-iio gr-iqbal gr-limesdr (U) gr-osmosdr gr-rds quisk (U) uhd Adam Borowski btrfs-progs Agustin Henze logbook Alan Boudreault mapserver (U) Alastair McKinstry ecflow pyferret xdmf Anders Waananen nordugrid-arc (U) Andreas Bombe gr-limesdr (U) soapysdr (U) Andreas Metzler hugin (U) libvigraimpex (U) Andreas Tille atropos (U) conda-package-handling (U) epigrass (U) libsbml (U) obitools (U) python-thinc (U) umis (U) Andrew Bartlett samba (U) Andrius Merkys openbabel (U) Anthony Wong pycangjie (U) Anton Gladky python-demgengeo (U) Aron Xu ukui-menus (U) Axel Beckert gnudatalanguage (U) Balint Reczey libcec (U) Barak A. Pearlmutter mlpack (U) Bas Couwenberg mapserver (U) qgis (U) Bastien Roucariès pythonmagick (U) Benjamin Drung rdma-core Bernd Zeimetz ceph (U) Boyuan Yang libplist (U) Carl Fürstenberg pythonmagick (U) Carsten Schoenert kicad (U) kopanocore (U) Ceph Packaging Team ceph Christoph Berg gr-limesdr (U) gr-soapy (U) postgresql-multicorn (U) quisk (U) Christoph Egger fife (U) python-enet Christopher Schramm blueman Daniel Kahn Gillmor fontforge (U) Daniel Leidert openbabel (U) Danny Edel borgbackup (U) Davide Viti fontforge (U) Debian 3D-Printer Packaging Team <3dprinter-gene...@lists.alioth.debian.org> printrun Debian Astronomy Team astrometry.net gnudatalanguage Debian Borg Collective borgbackup Debian DNS Team ldns Debian Edu Packaging Team sdaps Debian Electronics Team kicad Debian Fonts Task Force fontforge Debian Fonts Task Force psautohint Debian Games Team cegui-mk2 fife Debian GIS Project mapserver qgis saga Debian GNOME Maintainers glom libpwquality Debian Hamradio Maintainers gr-fcdproplus gr-gsm gr-limesdr gr-soapy quisk soapysdr Debian Input Method Team pycangjie Debian LibreOffice Maintainers libixion liborcus Debian Med Packaging Team atropos biosig4c++ conda-package-handling epigrass gdcm insighttoolkit4 libsbml obitools pymia simpleitk umis vtk-dicom Debian Multimedia Maintainers csound libopenshot openvdb Debian PhotoTools Maintainers hugin opencolorio openimageio Debian PostgreSQL Maintainers postgresql-multicorn Debian Printing Team hplip Debian Python Modules Team portio pyodbc (U) pythonmagick Debian QA Group link-grammar Debian Samba Maintainers ldb samba talloc tdb Debian Science Maintainers caffe libvigraimpex mlpack netgen openturns orocos-kdl python-thinc ros-geometry2 ros-image-common ros-ros-comm ros-rviz ros-vision-opencv siconos veusz Debian Science Team apertium apertium-lex-tools apriltag cg3 cryptominisat getfem++ lttoolbox morse-simulator neuron opencv plplot python-demgengeo sagemath vtk7 Debian Security Tools libbde libesedb libevt libevtx libewf libfsapfs libfsntfs libfvde libfwnt libfwsi liblnk libmsiecf libolecf libpff libqcow libregf libscca libsigscan libsmdev libsmraw libvhdi libvmdk libvshadow libvslvm Debian SSSD Team sssd Debian SSSD Team pam-wrapper Debichem Team avogadrolibs chemps2 openbabel rdkit Deepak Tripathi pyodbc Denis Barbier openturns (U) Didier Raboud hplip (U) Dima Kogan apriltag (U) Dmitry Smirnov
Re: Python3 modules not built for all supported Python versions
On 3/30/20 1:24 PM, Emilio Pozuelo Monfort wrote: > Hi, > > We've just finished the transition to python3.8 as the default python3 > interpreter, which was a bit difficult due to some autopkgtest regressions in > a > few rdeps, and to the fact that many modules only build their extensions for > the > default python version, which means they have a strict dependency on the > python3 > version[1] and they need to be rebuilt and migrated in lockstep with > python3-defaults. > > I have analyzed this based on current sid amd64 contents and have come up with > the following packages that don't ship extensions for both py3.7 and 3.8 > (which > are the currently supported versions). Note that pure python packages that > don't > build C extensions are not affected. > > It would be great if this situation can be improved in order to help with > future > python transitions. Building for all the supported python versions can be done > by build-depending on python3-all-dev and compiling your package (or just the > python bits) with PYTHON pointing to each version. Depending on your package's > build system, this could be largely automated using some helper, such as > pybuild. If you don't know how to add support for your package, feel free to > ask. Yes, that would be useful. However there are only limited time windows when you can do *and* check such work, i.e. when the python3-defaults package has two supported python3 versions. Filing and fixing these issues would be nice. Plus for most of those packages you have to modify the build system to build the package for each python3 version, or to just build the extension for the non-default version. I assume we still could keep 3.7 as a supported python3 version for some time, if people want to work on those issues. But people probably won't have the time to work on that when we introduce 3.9. Matthias
Re: Python3 modules not built for all supported Python versions
Hi, Quoting Emilio Pozuelo Monfort (2020-03-30 13:24:01) > We've just finished the transition to python3.8 as the default python3 > interpreter, which was a bit difficult due to some autopkgtest regressions in > a few rdeps, and to the fact that many modules only build their extensions > for the default python version, which means they have a strict dependency on > the python3 version[1] and they need to be rebuilt and migrated in lockstep > with python3-defaults. > > I have analyzed this based on current sid amd64 contents and have come up with > the following packages that don't ship extensions for both py3.7 and 3.8 > (which > are the currently supported versions). Note that pure python packages that > don't > build C extensions are not affected. > > It would be great if this situation can be improved in order to help with > future > python transitions. Building for all the supported python versions can be done > by build-depending on python3-all-dev and compiling your package (or just the > python bits) with PYTHON pointing to each version. Depending on your package's > build system, this could be largely automated using some helper, such as > pybuild. If you don't know how to add support for your package, feel free to > ask. does this mean that build-depending on python3-dev is wrong in general and should instead be replaced by build-depending on python3-all-dev? Could wrong build dependencies or binary packages which have a strict dependency on a python3 version not easily be detected by lintian? I'm not an expert in Python module packaging -- it just so happens that some packages I maintain offer Python bindings. Do you have a diff of a source package that successfully made the transition so that I know what I would have to change? For example the package src:ros-geometry2 has a super simple dh-style rules file, basically just doing: %: dh $@ --buildsystem=cmake --with python3 What would I have to change to successfully fix this problem? Thanks! cheers, josch signature.asc Description: signature
Re: Python3 modules not built for all supported Python versions
On Mon, 30 Mar 2020 at 15:30:01 +0200, Johannes Schauer wrote: > does this mean that build-depending on python3-dev is wrong in general and > should instead be replaced by build-depending on python3-all-dev? It is only wrong for packages that build Python 3 extensions (binary modules) that are intended to be loadable by all supported Python 3 versions (roughly: `find /usr/lib/python3/dist-packages -name '*.so'`). For packages that embed Python 3, like the versions of vim that have Python scripting support, or packages that use a Python 3 extension as an internal implementation detail of some tool, like gobject-introspection, my understanding is that build-depending on python3-dev continues to be appropriate. These extensions would ideally be installed in a private directory, like gobject-introspection's /usr/lib/x86_64-linux-gnu/gobject-introspection/giscanner/_giscanner.cpython-38-x86_64-linux-gnu.so - but I know some upstreams and some downstream maintainers (arguably incorrectly) package private extensions as though they were public extensions, because the mechanics of doing so are much simpler. > For example the package src:ros-geometry2 has a super simple > dh-style rules file, basically just doing: > > %: > dh $@ --buildsystem=cmake --with python3 > > What would I have to change to successfully fix this problem? The general answer is that you would have to build it repeatedly in a loop, with each supported version of Python 3 in turn. I am not aware of a way to do this in a similarly simple rules file. smcv
Re: Python3 modules not built for all supported Python versions
On 30/03/2020 16:08, Simon McVittie wrote: > On Mon, 30 Mar 2020 at 15:30:01 +0200, Johannes Schauer wrote: >> does this mean that build-depending on python3-dev is wrong in general and >> should instead be replaced by build-depending on python3-all-dev? > > It is only wrong for packages that build Python 3 extensions (binary > modules) that are intended to be loadable by all supported Python > 3 versions (roughly: `find /usr/lib/python3/dist-packages -name '*.so'`). > > For packages that embed Python 3, like the versions of vim that > have Python scripting support, or packages that use a Python 3 > extension as an internal implementation detail of some tool, like > gobject-introspection, my understanding is that build-depending > on python3-dev continues to be appropriate. These extensions would > ideally be installed in a private directory, like gobject-introspection's > /usr/lib/x86_64-linux-gnu/gobject-introspection/giscanner/_giscanner.cpython-38-x86_64-linux-gnu.so > - but I know some upstreams and some downstream maintainers (arguably > incorrectly) package private extensions as though they were public > extensions, because the mechanics of doing so are much simpler. > >> For example the package src:ros-geometry2 has a super simple >> dh-style rules file, basically just doing: >> >> %: >> dh $@ --buildsystem=cmake --with python3 >> >> What would I have to change to successfully fix this problem? > > The general answer is that you would have to build it repeatedly in a > loop, with each supported version of Python 3 in turn. I am not aware > of a way to do this in a similarly simple rules file. I've heard pybuild now has a cmake backend, so theoretically you could do something like %: dh $@ --buildsystem=pybuild --system=cmake Since I couldn't find any package in the archive that used pybuild with the cmake backend, I have looked at this specific package and came up with the attached debdiff. I added the dh_auto_install hack because the cmake build system creates the python extension as _tf2.so, and is only renamed by dh_python3 (as can be seen in current build logs). However when building for both python3.7 and python3.8, the last installation will override the former, and only one _tf2.so will be available. However that workaround proved to not be enough, because dh_install3 renames both the 3.7 and 3.8 versions as 3.8: make[2]: Leaving directory '/build/ros-geometry2-0.6.6/.pybuild/cpython3_3.7/build' I: pybuild pybuild:310: dh_python3 I: dh_python3 fs:343: renaming _tf2.so to _tf2.cpython-38-x86_64-linux-gnu.so [...] make[2]: Leaving directory '/build/ros-geometry2-0.6.6/.pybuild/cpython3_3.8/build' I: pybuild pybuild:310: dh_python3 W: dh_python3 fs:340: destination file exist, cannot rename _tf2.so to _tf2.cpython-38-x86_64-linux-gnu.so Resulting in these installed files: -rw-r--r-- root/root 69112 2020-03-30 14:56 ./usr/lib/python3/dist-packages/tf2_py/_tf2.cpython-38-x86_64-linux-gnu.so -rw-r--r-- root/root 69128 2020-03-30 14:56 ./usr/lib/python3/dist-packages/tf2_py/_tf2.so I don't know if I'm missing an argument to dh_python3 so that it knows the python version, or even if there's a better workaround. But perhaps pybuild should be doing this automatically between the dh_auto_install calls so that this kind of workarounds aren't necessary. Piotr, is this a bug in pybuild, or am I doing something wrong? Cheers, Emilio diff -Nru ros-geometry2-0.6.6/debian/changelog ros-geometry2-0.6.6/debian/changelog --- ros-geometry2-0.6.6/debian/changelog2020-01-18 21:51:17.0 +0100 +++ ros-geometry2-0.6.6/debian/changelog2020-03-30 16:56:18.0 +0200 @@ -1,3 +1,10 @@ +ros-geometry2 (0.6.6-1.1) UNRELEASED; urgency=medium + + * Non-maintainer upload. + * Build for all supported versions of python3. + + -- Emilio Pozuelo Monfort Mon, 30 Mar 2020 16:56:18 +0200 + ros-geometry2 (0.6.6-1) unstable; urgency=medium * New upstream version 0.6.6 diff -Nru ros-geometry2-0.6.6/debian/control ros-geometry2-0.6.6/debian/control --- ros-geometry2-0.6.6/debian/control 2020-01-18 21:51:17.0 +0100 +++ ros-geometry2-0.6.6/debian/control 2020-03-30 16:56:18.0 +0200 @@ -6,7 +6,7 @@ Leopold Palomo-Avellaneda Build-Depends: debhelper-compat (= 12), dh-exec, catkin (>= 0.7.14-4), libroscpp-core-dev, ros-message-generation, libstd-msgs-dev, - python3-dev, python3-setuptools, dh-python, + python3-all-dev, python3-setuptools, dh-python, libgeometry-msgs-dev, ros-actionlib-msgs, libconsole-bridge-dev, python3-rospy (>= 1.14.3+ds1-7), python3-rosgraph (>= 1.14.3+ds1-7), libactionlib-dev, librosconsole-dev, diff -Nru ros-geometry2-0.6.6/debian/rules ros-geometry2-0.6.6/debian/rules --- ros-geometry2-0.6.6/debian/rules2019-10-25 17:04:01.0 +0200 +++ ros-geometry2-0.6.6/debian/rules2020-03-30 16:56:18.0 +0200 @@
Re: Python3 modules not built for all supported Python versions
> I don't know if I'm missing an argument to dh_python3 so that it knows the > python version, or even if there's a better workaround. But perhaps pybuild better workaround is to force cmake to install into versioned dist-packages (that's what I do in distutils) as I didn't find a reliable way to read Python version/architecture/tag/etc. from .so file > should be doing this automatically between the dh_auto_install calls so that > this kind of workarounds aren't necessary. it's actually a nice idea - pybuild can check if /usr/lib/python3.X/dist-packages/ is empty and there are .so files in /usr/lib/python3/dist-packages/ and move them temporarily to /usr/lib/python3.X/dist-packages/ (it knows which interpreter is used at this point) and let dh_python3 rename them later > Piotr, is this a bug in pybuild, or am I doing something wrong? well, cmake is not really easy to work with, they provide -DPYTHON_LIBRARY:FILEPATH… but they don't use it or use completely different name… it basically is different for each library I checked (and in different official documentation versions I read, sic!)
re: samba: FTBFS glibc/stropts/python issue.
The samba FTBFS is blocking the python 3.8 transition in raspbian bullseye, so I decided to take a look. error: Unable to determine origin of type `struct cli_credentials' I don't think this is the error that is causing the build failure. The same error can be seen in succesful build logs. e.g. https://buildd.debian.org/status/fetch.php?pkg=samba&arch=amd64&ver=2%3A4.11.5%2Bdfsg-1%2Bb1&stamp=1583775414&raw=0 Instead I think the real error is further down the log ../../lib/replace/system/network.h:91:10: fatal error: stropts.h: No such file or directory Some googling lead me to https://bugs.gentoo.org/699668 and https://bugzilla.samba.org/show_bug.cgi?id=14100 which suggests that the bug triggers if samba is built against a newer glibc while python was built against an older glibc. Specifically it seems python headers leak certain system feature defines including HAVE_STROPTS_H which cause network.h to think stropts.h is available when it isn't. If my understanding is correct I see two possible ways forward. 1. Rebuild python3.8 against the new glibc. 2. Remove the stropts include from samba, it doesn't seem to actually be used for anything (at least I can't find any other references to HAVE_STROPTS_H in the code). I am currently testing a build in raspbian bullseye with the second approach. I will report back later on whether it results in a succesful build.
py2removal: proposal to increase apps popcon limit
Hello, as part of the py2removal process, we're not raising the bug priority to serious if it's associated with an application and popcon is bigger than 300 There are currently: * 38 apps that are leaf packages and popcon > 300; ** 13 out of 38 have popcon < 600 ** 20 out of 38 have popcon < 1000 I propose to raise the popcon threshold to 1000. What are your thoughts? I will move forward with this change in 3 days if i dont hear any opposition. Cheers, -- Sandro "morph" Tosi My website: http://sandrotosi.me/ Me at Debian: http://wiki.debian.org/SandroTosi Twitter: https://twitter.com/sandrotosi
Re: samba: FTBFS glibc/stropts/python issue.
If my understanding is correct I see two possible ways forward. 1. Rebuild python3.8 against the new glibc. 2. Remove the stropts include from samba, it doesn't seem to actually be used for anything (at least I can't find any other references to HAVE_STROPTS_H in the code). I am currently testing a build in raspbian bullseye with the second approach. I will report back later on whether it results in a succesful build. This build was successful and I uploaded it to raspbian, a debdiff should appear soon at https://debdiffs.raspbian.org/main/s/samba . No intent to NMU in Debian.
Re: Bug#954582: samba: FTBFS glibc/stropts/python issue.
On Tue, 2020-03-31 at 02:09 +0100, peter green wrote: > The samba FTBFS is blocking the python 3.8 transition in raspbian > bullseye, so I decided to take a look. Thanks for digging into this. I've updated the Samba bug below, and I think this is a Python bug and needs to be fixed in Python. The correct fix is for Python not to leak these defines. Andrew Bartlett -- Andrew Bartlett https://samba.org/~abartlet/ Authentication Developer, Samba Team https://samba.org Samba Developer, Catalyst IT https://catalyst.net.nz/services/samba
