Re: Numpy API change?
Hi all, On Tue, Jul 20, 2010 at 15:07, Jakub Wilk wrote: > * Antoine Sirinelli , 2010-07-19, 21:27: >> >> It seems that since the last numpy upload, pytables is no longer working. >> I >> have filled a bug report >> (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=589669) but my question >> is >> more general. It seems that pytables is broken because of a change in >> numpy >> API. Is there a way to avoid that using good dependences in the different >> deb >> packages? >> >> I may be completely wrong about the origin of this bug. Sorry if it is the >> case. > > Size of numpy.dtype has grown in Numpy 1.4 (some new fields has been > appended to the end of the structure). This itself does not constitute an > ABI breakage; however, some Python extensions[1] are checking if size of > imported types is exactly the same as it were at build time. Therefore, they > won't work if built with Numpy 1.3 and run with Numpy 1.4 (or the other way > round). > > Some other extensions[2] are more lax when it comes to comparing type sizes, > and they crash only if the size shrank since build time. > > [1] At least these: > - python-tables > - mayavi2 > Rebuilding their Pyrex/Cython sources with recent Cython should move them > into the second group. > > [2] At least these: > - python-nipy-lib > - python-petsc4py - python-scikits-learn-lib > - python-scipy We just had a chat on #debian-python in order to find a way to avoid this situation to happen again in the future. Fact is, we should find a way to tight the dependencies on python-numpy, for example we can re-inforce some sort of "python-numpt (>= MAJOR.MINOR), python-numpy (<< MAJOR.MINOR+1)" (where MAJOR is the current major version number, and MINOR is the minor one). That means that at each update of major/minor release of numpy, the depending packages needs to be binNMU'ed or receive a sourceful upload. We also need to find some way to automate this versioned information addition and the following depends on that by depending packages. Piotr developed a nice approach when writing dh_python2 (and promised to backport it to python-support too): he introduced the pydist concept. This is a way to express in the current package A what are the dependencies other packages B, C, ... should use when depending on A. (the example is not strictly what will be in reality, it's just a mock) Let's say python-a declares a pydist of "python-a << (x.y)", then python-b can call the helper like "dh_python2 --with-pydist=A" and have the dependencies on "python-a << (x.y)" automatically added to its Depends line. Now, this is very nice, but it's still not a solution we can use it right now. Instead, what we can do is to prepare the ground to make this happen when we're ready. So what I think we can do right now is: - for the time being, declare a (build-)depends on python-numpy >= 1.4.1 in rdeps of python-numpy - move any debian packages depending on python-numpy to use python-support; if you want help to do it (either be some basic guidelines or a complete&working patch), just contact me: I already moved several package from central to support, so I know how to do that and I'd be happy to help fixing part of this mess (that incidentally I generated uploading 1.4.1 :) ). Please express any thoughts or dubts, I'd be happy to hear them and answer if I can :) If I missed some packages in the loop, please forward it to them too. Cheers, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/aanlktimjdpnwlobyjt2soycrjav1dvheccmd4yae7...@mail.gmail.com
Re: Numpy API change?
On Wed, 21 Jul 2010, Sandro Tosi wrote: > Fact is, we should find a way to tight the dependencies on > python-numpy, for example we can re-inforce some sort of "python-numpt > (>= MAJOR.MINOR), python-numpy (<< MAJOR.MINOR+1)" (where MAJOR is the > current major version number, and MINOR is the minor one). That means > that at each update of major/minor release of numpy, the depending > packages needs to be binNMU'ed or receive a sourceful upload. Pardon possible ignorance -- but isn't all the situation with transition from numpy 1.3 to 1.4 is actually the effect of absent proper transition in Debian for numpy from 1.3 to 1.4? Restricting version might be an overkill as a generic solution since most of the modules (and may be extensions) would not need that and we do not precisely know what will happen in 1.5 to restrict just some packages. So how can we do transitions without involving maintainers for all dependent projects in regard to their knowledge about compatibility with upcoming new version? Due to peculiarities of Python it is harder to assure compatibility even of pure Python modules without extensions with a transition to a new version, even at pure API level unless there is 100% code coverage by unittests. But assuming that in longer run we agree on how we invoke unittesting for Python modules we ship, then for each future transition all dependent modules could easily be tested to unveal bad cows: if unittests (against installed version) are ok: pass elif re-build (including unittests against re-built version) is ok: binNMU else: ring the bell Or am I simply hallucinating and missing the point? -- .-. =-- /v\ = Keep in touch// \\ (yoh@|www.)onerussian.com Yaroslav Halchenko /( )\ ICQ#: 60653192 Linux User^^-^^[17] signature.asc Description: Digital signature
Re: Numpy API change?
[Taking *...@packages.qa.debian.org off Cc, in hope that everybody reads d-pyt...@...] * Yaroslav Halchenko , 2010-07-21, 16:17: Fact is, we should find a way to tight the dependencies on python-numpy, for example we can re-inforce some sort of "python-numpt (>= MAJOR.MINOR), python-numpy (<< MAJOR.MINOR+1)" (where MAJOR is the current major version number, and MINOR is the minor one). That means that at each update of major/minor release of numpy, the depending packages needs to be binNMU'ed or receive a sourceful upload. Pardon possible ignorance -- but isn't all the situation with transition from numpy 1.3 to 1.4 is actually the effect of absent proper transition in Debian for numpy from 1.3 to 1.4? It's true that upload of Numpy 1.4 wasn't done with proper care. However, even if we discovered the breakage earlier, we couldn't avoid sourceful uploads of the affected packages. each future transition all dependent modules could easily be tested to unveal bad cows: if unittests (against installed version) are ok: pass elif re-build (including unittests against re-built version) is ok: binNMU else: ring the bell BinNMUs only make sense if there is a versioned dependency that will force upgrades and migration in the correct order. We cannot solve the current situation that way: e.g. if we binNMUed python-tables, its new binaries would immediately migrate to testing, which would make them as useless as they are currently in unstable. -- Jakub Wilk signature.asc Description: Digital signature
Re: Numpy API change?
On Wed, Jul 21, 2010 at 22:17, Yaroslav Halchenko wrote: > Restricting version might be an overkill as a generic solution > since most of the modules (and may be extensions) would not need that > and we do not precisely know what will happen in 1.5 to restrict just > some packages. the idea is to restrict versions only on the packages that needs it, where the maintaienr will use the --with-pydist=python-foo and have the Depends on python-foo generated for him; if it's just an "easy" depends on foo, just put python-foo in Depends and don't call --with-pydist -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/aanlktimw82mrzuwqkhjhw74pvzefkdppnvklfiq4a...@mail.gmail.com
