Re: [Python-modules-team] Bug#643873: Making Numpy transition less painful

2011-10-31 Thread Sandro Tosi
Hello Jakub,
sorry for this very late reply.

I'm adding in the loop debian-python, so that we can hear others
opinion (and pretend we're still a team :) ).

First of all, let me say: I like your idea! in the reply (full, for
new readers) my comments.

On Fri, Sep 30, 2011 at 17:17, Jakub Wilk  wrote:
> Source: python-numpy
> Version: 1:1.5.1-2
> Severity: normal
>
> Numpy transitions are currently very painful: they involve large number of
> packages, and they need to happen even when Numpy *didn't* break binary
> compatibility. Here's my proposal to improve this situation.
>
> Virtual packages
> 
> python-numpy would provide two virtual packages:
> - python-numpy-abi$N and
> - python-numpy-api$M.
> $N would be changed each time C_ABI_VERSION is changed upstream, and $M
> would be changed each time C_API_VERSION is changed upstream.
>
> Dependencies generated by dh_numpy
> ==
> dh_numpy would generate dependency either on "python-numpy-abi$N" (by
> default) or on "python-numpy-api$N" (if a special option, say, --strict, is
> used). The second option would be needed for packages which are overzealous
> at checking ABI unless/until we fix them (we have at least one package in
> the archive which does that).

do you think it's possible for the same source package to generate 2
binaries one needing python-numpy-abi$N and the other needing
python-numpy-api$N ? how would you imagine to handle this? with
"dh_numpy -pbin1 --strict\ndh_numpy -pbin2" or something else?

> Avoiding strict dependencies for arch:all packages
> ==
> dh_numpy should not generate any dependencies in arch:all packages. Such
> packages cannot rely on Numpy ABI, and rebuilding them for transition
> purposes is causes lot of trouble. (We already had to do this in the past!)
> For the same reason, /usr/share/python/dist/python-numpy needs to go away.

so just a python-numpy added to python:Depends? how does it fix with a
source that generates an arch:all and an arch:any (the latter using
either -abi or -api)?

> Open problems
> =
>
> Virtual package names
> -
> How should $N and $M be generated? At least C_ABI_VERSION seems to be a big
> integer, at it'd look ugly if we used is an $N. :/

Yeah, I can talk about that with upstream, in particularly because
maybe they only increment the right-most digit and don't do big jumps.
Else, we can just start with 1 for both and use it only for Debian
purpose.

> Gracefully handling the upcoming transition
> ---
> A number of packages in the archive already has dependency on "python-numpy
> (<< 1:1.6)", which is not satisfiable by the package in experimental. To
> avoid this transition (or at least reduce its size) we could do one of two
> things:
> 1) Implement the proposed changes first in unstable, schedule binNMUs, then
> upload Numpy 1.6 to unstable.
> 2) Implement the proposed changes only for Numpy 1.6, but make dh_numpy
> generate alternative versioned dependencies:
> "python-numpy (>= 1:1.5.1), python-numpy-abi$N | python-numpy (<< 1:1.6.2)".
> Then upload Numpy 1.6, schedule binNMUs.

Mh, my lazy instinct would say 2) that has also the advantage to bring
1.6 into unstable (but I guess 1.7 is coming soon).

Please also note taht my perl foo is barely None, so I'd need some
help in extending dh_numpy :)

Cheers,
-- 
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/CAPdtAj0ce-M8t+pYiZjxz-5PL5e=-ec2yhataphqrhab6-2...@mail.gmail.com



Re: [Python-modules-team] Bug#643873: Making Numpy transition less painful

2011-10-31 Thread Jakub Wilk

* Sandro Tosi , 2011-10-31, 19:57:

Dependencies generated by dh_numpy
==
dh_numpy would generate dependency either on "python-numpy-abi$N" (by 
default) or on "python-numpy-api$N" (if a special option, say, 
--strict, is used).


I just realized that, in order to make packages using Cython happy, the 
first one will have to be:


"python-numpy-abi$N, python-numpy (>= $V)"

where $V is the first version with the current API. [0]

The second option would be needed for packages which are overzealous 
at checking ABI unless/until we fix them (we have at least one package 
in the archive which does that).


[1]

do you think it's possible for the same source package to generate 2 
binaries one needing python-numpy-abi$N and the other needing 
python-numpy-api$N ?


It'd quite crazy, but in theory, yes, that could happen.

how would you imagine to handle this? with "dh_numpy -pbin1 
--strict\ndh_numpy -pbin2"


Exactly.


Avoiding strict dependencies for arch:all packages
==
dh_numpy should not generate any dependencies in arch:all packages. 
Such packages cannot rely on Numpy ABI, and rebuilding them for 
transition purposes is causes lot of trouble. (We already had to do 
this in the past!) For the same reason, 
/usr/share/python/dist/python-numpy needs to go away.


so just a python-numpy added to python:Depends?


I was rather thinking about not touching such packages at all. But 
adding unversioned python-numpy to ${python:Depends} would work, too.


how does it fix with a source that generates an arch:all and an 
arch:any (the latter using either -abi or -api)?


dh_numpy would generate different substitutions for each of them. I 
don't see any problem here.



Gracefully handling the upcoming transition
---
A number of packages in the archive already has dependency on 
"python-numpy (<< 1:1.6)", which is not satisfiable by the package in 
experimental. To avoid this transition (or at least reduce its size) 
we could do one of two things:
1) Implement the proposed changes first in unstable, schedule binNMUs, 
then upload Numpy 1.6 to unstable.
2) Implement the proposed changes only for Numpy 1.6, but make 
dh_numpy generate alternative versioned dependencies:

"python-numpy (>= 1:1.5.1), python-numpy-abi$N | python-numpy (<< 1:1.6.2)".
Then upload Numpy 1.6, schedule binNMUs.


Mh, my lazy instinct would say 2) that has also the advantage to bring
1.6 into unstable (but I guess 1.7 is coming soon).


Yeah, that option sounded very appealing, but turned out not feasible 
after all. See my comment about Cython at the top of this mail.  
C_API_VERSION has been bumped from 4 to 6, so $V should be at least >= 
1:1.6.


Please also note taht my perl foo is barely None, so I'd need some help 
in extending dh_numpy :)


No problem, I can do the coding part once we agree how everything should 
work. :)



[0] Cython generates this code for importing C types:

|if (!strict && ((PyTypeObject *)result)->tp_basicsize > (Py_ssize_t)size) {
|PyOS_snprintf(warning, sizeof(warning),
|"%s.%s size changed, may indicate binary incompatibility",
|module_name, class_name);
|#if PY_VERSION_HEX < 0x0205
|if (PyErr_Warn(NULL, warning) < 0) goto bad;
|#else
|if (PyErr_WarnEx(NULL, warning, 0) < 0) goto bad;
|#endif
|}
|else if (((PyTypeObject *)result)->tp_basicsize != (Py_ssize_t)size) {
|PyErr_Format(PyExc_ValueError,
|"%s.%s has the wrong size, try recompiling",
|module_name, class_name);
|goto bad;
|}

That is, import will fail if the size imported type is shorter at 
runtime that at build time.



[1] Pyrex (and very old versions of Cython) generates this code for 
importing C types:


| if (((PyTypeObject *)result)->tp_basicsize != size) {
| PyErr_Format(PyExc_ValueError,
| "%s.%s does not appear to be the correct type object",
| module_name, class_name);
| goto bad;

So import will fail with exception if the size imported type is 
different at runtime that at build time.


--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20111031210725.ga4...@jwilk.net