Re: dh_python and python policy analysis

[Steve Langasek]

On Sat, Aug 12, 2006 at 12:10:07PM -0500, Manoj Srivastava wrote:

> >> 3.1.3. Provides
> >> 
> >> Packages with public modules and extensions should be named, or
> >> should provide, python-foo. Pure Python public modules that support
> >> all Python versions need not have a Provides field.

> > ... unless there is an application using a non-default python
> > version using this module. or else you require the application
> > depending on any indirect dependency of python-foo.

> Hmm.  Two things: if application X requires my pure python
>  public module (called, say, python-foo), and uses some specific
>  version of python, why can't it depend on just python-foo Why do I
>  have to provide pythonX.Y-foo?

Because a dependency on "python-foo" expresses the request "give me the foo
module for the current version of python".  There is no guarantee that the
python-foo package installed is compatible with, or provides support for,
the pythonX.Y you're using, except if this package declares a Provides:
pythonX.Y-foo; so the depends/provides: pythonX.Y-foo needs to be there to
ensure that the app and the modules it needs aren't allowed to get out of
sync on a user's system (or in testing).

> Also, as a maintainer of python-foo, I can't know when such an
>  application would be created, and  we are trying to minimize
>  reuploads of packages -- so either one provides all such
>  pythonX.Y-foo at the get go, and reupload at every new python version
>  or dropping of the old version -- or we upload every time some app is
>  uploaded that may require yet abother X.y, and when we drop a version
>  of Python.

Such apps would ideally be few and far between, but after thinking about it
for a while, I wasn't actually able to come up with a concrete case where
having the Provides: declared ahead of time complicates transitions more
than not having them would.  For pure python modules, this still means
inconvenient sourceful reuploads when new python implementations become
available, since the Provides: can't be declared for pythonX.Y that we don't
yet know about, but fortunately those reuploads would only need to be done
on demand for modules that are actually used from scripts invoking a
non-default python interpreter.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
[EMAIL PROTECTED]   http://www.debian.org/


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: dh_python and python policy analysis

[Manoj Srivastava]

On Sun, 13 Aug 2006 00:01:37 -0700, Steve Langasek <[EMAIL PROTECTED]> said: 

> On Sat, Aug 12, 2006 at 12:10:07PM -0500, Manoj Srivastava wrote:
>> >> 3.1.3. Provides
>> >> 
>> >> Packages with public modules and extensions should be named, or
>> >> should provide, python-foo. Pure Python public modules that
>> >> support all Python versions need not have a Provides field.

>> > ... unless there is an application using a non-default python
>> > version using this module. or else you require the application
>> > depending on any indirect dependency of python-foo.

>> Hmm.  Two things: if application X requires my pure python public
>> module (called, say, python-foo), and uses some specific version of
>> python, why can't it depend on just python-foo Why do I have to
>> provide pythonX.Y-foo?

> Because a dependency on "python-foo" expresses the request "give me
> the foo module for the current version of python".

No, it does not. When a package bar depend on package baz, it
 means just that-- that it requires  the package baz to work. With the
 policy draft that I have proposed, if it is a pure python module,
 with no intrinsic restrictions on the versions of python supported,
 other packages can just depend on the package, knowing that a policy
 compliant package would support all available versions.

> There is no guarantee that the python-foo package installed is
> compatible with, or provides support for, the pythonX.Y you're
> using, except if this package declares a Provides: pythonX.Y-foo; so
> the depends/provides:

Rubissh. You are just making up these rules, and since it
 hurts, just don't make them up.  If you want a pure python module
 that complies with the new policy, and does not provide
 pythonX.Y-foo; you know trhat you can just depend on python-foo, and
 things shall work.

> pythonX.Y-foo needs to be there to ensure that the app and the
> modules it needs aren't allowed to get out of sync on a user's
> system (or in testing).

And why would they get out of sync?  If they are compliant,
 then when a new python version is installed the module is compiled
 for it -- so no matter what version of python you use, there is a
 compiled .pyc file there.

>> Also, as a maintainer of python-foo, I can't know when such an
>> application would be created, and we are trying to minimize
>> reuploads of packages -- so either one provides all such
>> pythonX.Y-foo at the get go, and reupload at every new python
>> version or dropping of the old version -- or we upload every time
>> some app is uploaded that may require yet abother X.y, and when we
>> drop a version of Python.

> Such apps would ideally be few and far between, but after thinking
> about it for a while, I wasn't actually able to come up with a
> concrete case where having the Provides: declared ahead of time
> complicates transitions more than not having them would.  For pure
> python modules, this still means inconvenient sourceful reuploads
> when new python implementations become available, since the
> Provides: can't be declared for pythonX.Y that we don't yet know
> about, but fortunately those reuploads would only need to be done on
> demand for modules that are actually used from scripts invoking a
> non-default python interpreter.

And if you just follow the new policy, no uploads are needed
 at all. The new policy wins.

manoj
-- 
America: born free and taxed to death.
Manoj Srivastava   <[EMAIL PROTECTED]>  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: dh_python and python policy analysis

[Steve Langasek]

On Sun, Aug 13, 2006 at 03:32:27AM -0500, Manoj Srivastava wrote:
> On Sun, 13 Aug 2006 00:01:37 -0700, Steve Langasek <[EMAIL PROTECTED]> said: 

> > On Sat, Aug 12, 2006 at 12:10:07PM -0500, Manoj Srivastava wrote:
> >> >> 3.1.3. Provides

> >> >> Packages with public modules and extensions should be named, or
> >> >> should provide, python-foo. Pure Python public modules that
> >> >> support all Python versions need not have a Provides field.

> >> > ... unless there is an application using a non-default python
> >> > version using this module. or else you require the application
> >> > depending on any indirect dependency of python-foo.

> >> Hmm.  Two things: if application X requires my pure python public
> >> module (called, say, python-foo), and uses some specific version of
> >> python, why can't it depend on just python-foo Why do I have to
> >> provide pythonX.Y-foo?

> > Because a dependency on "python-foo" expresses the request "give me
> > the foo module for the current version of python".

> No, it does not. When a package bar depend on package baz, it
>  means just that-- that it requires  the package baz to work. With the
>  policy draft that I have proposed, if it is a pure python module,
>  with no intrinsic restrictions on the versions of python supported,
>  other packages can just depend on the package, knowing that a policy
>  compliant package would support all available versions.

Then what do you name a package that is a pure python module that *DOES*
have intrinsic restrictions on the version of python supported?  You are
wrong to assume that a pure python module will automatically support all
available versions of python -- if all python versions were completely
backwards- and forwards-compatible with one another, there would be no
reason in the first place to *have* multiple versions of them in the
archive.

The reality is that there *are* language differences with each
implementation of python, and a pure python module may *not* work with any
given implementation of python available in the archive, and we need a way
to express such dependencies that guards users against package relationships
being satisfied by broken combinations.

> > There is no guarantee that the python-foo package installed is
> > compatible with, or provides support for, the pythonX.Y you're
> > using, except if this package declares a Provides: pythonX.Y-foo; so
> > the depends/provides:

> Rubissh. You are just making up these rules, and since it
>  hurts, just don't make them up.

What hurts is your ignorance of design requirements that were discussed at
length at the DebConf python BoF.

>  If you want a pure python module that complies with the new policy, and
> does not provide pythonX.Y-foo; you know trhat you can just depend on
> python-foo, and things shall work.

No, only packages that invoke /usr/bin/python as interpreter can do this.
Packages that invoke /usr/bin/pythonX.Y can't possibly have any forward
guarantee that python-foo will provide a working interface for the version
of pythonX.Y they have installed.

> > pythonX.Y-foo needs to be there to ensure that the app and the
> > modules it needs aren't allowed to get out of sync on a user's
> > system (or in testing).

> And why would they get out of sync. If they are compliant, then
> when a new python version is installed the module is compiled for it -- so
> no matter what version of python you use, there is a compiled .pyc file
> there.

Python decorators are a language feature available only in python 2.4 and
above.  Given a module foo version 1.0 which does not use decorators, a
module foo version 2.0 which does, and a package bar which invokes
/usr/bin/python2.3 and depends on "python2.3, python-foo", how do you
protect against installing python-foo 2.0 and breaking bar?  It doesn't
matter a bit whether the .pyc is in the python2.3 search path when the
module in question isn't compatible with that version of the language.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
[EMAIL PROTECTED]   http://www.debian.org/


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: dh_python and python policy analysis

[Manoj Srivastava]

On Sun, 13 Aug 2006 10:28:43 -0700, Steve Langasek <[EMAIL PROTECTED]> said: 

> On Sun, Aug 13, 2006 at 03:32:27AM -0500, Manoj Srivastava wrote:
>> On Sun, 13 Aug 2006 00:01:37 -0700, Steve Langasek
>> <[EMAIL PROTECTED]> said:

>> > On Sat, Aug 12, 2006 at 12:10:07PM -0500, Manoj Srivastava wrote:
>> >> >> 3.1.3. Provides

>> >> >> Packages with public modules and extensions should be named,
>> >> >> or should provide, python-foo. Pure Python public modules
>> >> >> that support all Python versions need not have a Provides
>> >> >> field.

>> >> > ... unless there is an application using a non-default python
>> >> > version using this module. or else you require the application
>> >> > depending on any indirect dependency of python-foo.

>> >> Hmm.  Two things: if application X requires my pure python
>> >> public module (called, say, python-foo), and uses some specific
>> >> version of python, why can't it depend on just python-foo Why do
>> >> I have to provide pythonX.Y-foo?

>> > Because a dependency on "python-foo" expresses the request "give
>> > me the foo module for the current version of python".

>> No, it does not. When a package bar depend on package baz, it means
>> just that-- that it requires the package baz to work. With the
>> policy draft that I have proposed, if it is a pure python module,
>> with no intrinsic restrictions on the versions of python supported,
>> other packages can just depend on the package, knowing that a
>> policy compliant package would support all available versions.

> Then what do you name a package that is a pure python module that
> *DOES* have intrinsic restrictions on the version of python
> supported?

WShy do you need a special name? Had you read the actual
 proposed draft, you'd realize that we do cater to that possibility.

> You are wrong to assume that a pure python module will automatically
> support all available versions of python -- if all python versions
> were completely backwards- and forwards-compatible with one another,
> there would be no reason in the first place to *have* multiple
> versions of them in the archive.

> The reality is that there *are* language differences with each
> implementation of python, and a pure python module may *not* work
> with any given implementation of python available in the archive,
> and we need a way to express such dependencies that guards users
> against package relationships being satisfied by broken
> combinations.

In these two paragraphs you are doing three things.
 a) Belabouring the obvious
 b) jumping to conclusions about what I assume,
 c) demonstrating that you have not read the proposed draft.

>> > There is no guarantee that the python-foo package installed is
>> > compatible with, or provides support for, the pythonX.Y you're
>> > using, except if this package declares a Provides: pythonX.Y-foo;
>> > so the depends/provides:

>> Rubissh. You are just making up these rules, and since it hurts,
>> just don't make them up.

> What hurts is your ignorance of design requirements that were
> discussed at length at the DebConf python BoF.

A little knowledge is dangerous. Go and educate yourself on
  what is being discussed in this thread -- or, if you are too
  almighty busy to familiarize yourself with the things you weigh in
  on, let meattempt to summarize, eliding any "oooh shiny" sections:

If you are packaging a public  pure python module (not
 extension or private modules), there are two cases: either you have
 intrinsic versions dependencies in your module, or not.

If you have version dependencies, your Depends and Provides
 rules are differennt, and you need to read the draft to see what they
 are.

If there are no such internal dependencies, then you need not
 provide pythonX.Y-foo virtual packages, since you should arrange for
 your packages to be compiled for any new version that is uploaded.

As to the BOF thing, I'll bite: Why one earth did the bof come
 up with design decisiosn that require  every single goldarned python
 module package to be reuploaded every time a new version of python is
 added or removed?

Why did the BOF choose to ignore the experience of the emacs
 listp community, that also has byte compilation issues, and subtle
 differences in byte compiler? Were the people in the bof aware that
 the python interpreter, like emacs itself, can optionally byte
 compile source on the fly, and lal that is lost is some speed?

Frankly, if the BOF came up with a design that requires every
 single public pure python module to be uploaded every single time a
 new version of python is added or removed, than you should be happy I
 am ignoring their delibrations.

Byte compiling interpred language files is not rocvket
 science. It is also not a new problem.

manoj
 beginning to realize why the state pof python in Debian is such a mess
-- 
To err is human, to moo bovine.
Manoj Srivastava   <[EMAIL PRO

Re: dh_python and python policy analysis

[Pierre Habouzit]

Le dim 13 août 2006 22:17, Manoj Srivastava a écrit :
>        As to the BOF thing, I'll bite: Why one earth did the bof come
>  up with design decisiosn that require  every single goldarned python
>  module package to be reuploaded every time a new version of python
> is added or removed?

actually,it's not truly needed, it's needed iff a package do needs the 
new provides or not.

and those reuploads are kind of binNMUs, the real problem here is that 
tehre is no arch:all binNMU and maybe that's here the problem that need 
fixing.
-- 
·O·  Pierre Habouzit
··O[EMAIL PROTECTED]
OOOhttp://www.madism.org


pgptHBQYsMOcp.pgp
Description: PGP signature

Re: dh_python and python policy analysis

[Manoj Srivastava]

Hi,

OK, I see I have to dot the i's and cross the t's for this
 case here.  So, here is the scenario: package python-foo packages a
 public pure python module.  Package bar imports the module
 foo. Package baz is a package not yet written that would be written
 for Python2.6 that would also need module foo, but only when
 we actually get python2.6. Let us also have a package bar-baz that is
 written for python2.5.

Also, ket us assume the module foo would work for versions
 2.3, 2.4, 2.5 -- but in the yet unreleased version 2.6, stuff
 changes, and module foo would not be compatible as written.  OK? 

State of Python at the start of the thought experiment: current
 is 2.3, available is also 2.4, and let us pretend no one has
 heard of 2.5 or 2.6 yet.

With me so far? 

Now we have two policy proposals, A, and B. A decrees that
 python-foo depends on python, ad has no provides. Policy B requires
 that python-foo also provide python2.3-foo and python2.4-foo.

The following transition events occur.

 1) Python 2.5 is added.
policy Apolicy B
no upload. python-foo recompiledUpload python-foo, adding
   for 2.5  provides for 2.5

No transition for package bar   package bar-baz must wait
or bar-baz  the upload.

 2) Default python version changes to 2,4  
 3) Python2.3 is dropped.
policy Apolicy B
no upload.  Upload python-foo, removing
provides for 2.3
 4) Python 2.6 is added.
   Here there are two cases. Either module foo can't be written at all
   for version 2.6, or it the same functionality can be provided with
   a code change, perhaps hidden behind a version conditional.

How often have people seen a regression in Python that
 something that was doable for version N can't be done at all in
 version N + 1?

 4a)  foo can be coded for version 2.6
 Policy A policy B
Package uploaded, with the changed   Package uploaded, with the
source. package baz has to wait  changed  source, and with
 the provides.  Package
 baz has to wait.
 4b) foo can't be written for version 2,6, or will take time, and
 support for 2,6 is dropped (at least for the moment)
 Policy A policy B
Package uploaded, with provides, Package uploaded, with provides
Packages bar, bar-baz, and baz   Package baz has to wait.
(rdepends python-foo) informed of
the provides. Need to upload the
rdepends

Now, most pure python packages will never see option 4 at all;
 and those that do, a number  will be case 4a.

Even for the case of 4b, there is time to do the transition
 for packages bar and bar-baz -- until 2.6 becomes the default, there
 is no critical bug in bar or bar-baz.

Now take this to every single pure python module package in
 Debian, multiply with the upload by default for every single
 addition or removal of python packages, and you can see that adding
 more work in the corner case 4b is worth not having to upload
 packages multiple times by default.

manoj
-- 
I used to be Snow White, but I drifted. Mae West
Manoj Srivastava   <[EMAIL PROTECTED]>  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: dh_python and python policy analysis

[Robert Collins]

On Sun, 2006-08-13 at 19:56 -0500, Manoj Srivastava wrote:
>Here there are two cases. Either module foo can't be written at all
>for version 2.6, or it the same functionality can be provided with


This is a little simplistic.

The parser changes fairly routinely in python versions. This means that
a version conditional is not sufficient to provide compatability with
older pythons - the module will not parse.

The usual thing done for cross version support is to write in the older
version of python, or in extreme cases (i.e. where performance really
hurts) have two separate modules _foo_2_5 and _foo_2_6 and conditionally
do
'from _foo_2_5 import *' etc.

-Rob
-- 
GPG key available at: .


signature.asc
Description: This is a digitally signed message part

Re: dh_python and python policy analysis

[Manoj Srivastava]

On Sun, 13 Aug 2006 23:37:15 +0200, Pierre Habouzit <[EMAIL PROTECTED]> said: 

> Le dim 13 août 2006 22:17, Manoj Srivastava a écrit :
>>       As to the BOF thing, I'll bite: Why one earth did the bof
>> come  up with design decisiosn that require  every single goldarned
>> python  module package to be reuploaded every time a new version of
>> python is added or removed?

> actually,it's not truly needed, it's needed iff a package do needs
> the new provides or not.

> and those reuploads are kind of binNMUs, the real problem here is
> that tehre is no arch:all binNMU and maybe that's here the problem
> that need fixing.

They are sourceful NMU's: since the  packaged source comprises
 of the .dsc, and that includes the full provides line, as well as the
 information in the source Packages file, etc, also needs to changes.

The draft does not require any more uploads in the default
 case than absolutely required.  I think the corner case that
 something that was feasible in the old version but not in th new one
 is a low probability event, really, so making that case a trifle
 inconvenient to ease the routine transitions of adding/removing
 python versions is a good tradeoff.

manoj
-- 
Novinson's Revolutionary Discovery: When comes the revolution, things
will be different -- not better, just different.
Manoj Srivastava   <[EMAIL PROTECTED]>  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: dh_python and python policy analysis

[Manoj Srivastava]

On Mon, 14 Aug 2006 11:21:07 +1000, Robert Collins <[EMAIL PROTECTED]> said: 

> On Sun, 2006-08-13 at 19:56 -0500, Manoj Srivastava wrote:
>> Here there are two cases. Either module foo can't be written at all
>> for version 2.6, or it the same functionality can be provided with

> This is a little simplistic.

> The parser changes fairly routinely in python versions. This means
> that a version conditional is not sufficient to provide
> compatability with older pythons - the module will not parse.

> The usual thing done for cross version support is to write in the
> older version of python, or in extreme cases (i.e. where performance
> really hurts) have two separate modules _foo_2_5 and _foo_2_6 and
> conditionally do 'from _foo_2_5 import *' etc.

Either conditional code or conditional inclusion are
 variations on the common theme I alluded to in my admittedly, and
 designedly, simplified example.

manoj
-- 
There's too much beauty upon this earth for lonely men to
bear. Richard Le Gallienne
Manoj Srivastava   <[EMAIL PROTECTED]>  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: dh_python and python policy analysis

[Steve Langasek]

On Sun, Aug 13, 2006 at 07:56:09PM -0500, Manoj Srivastava wrote:

> OK, I see I have to dot the i's and cross the t's for this
>  case here.  So, here is the scenario: package python-foo packages a
>  public pure python module.  Package bar imports the module
>  foo. Package baz is a package not yet written that would be written
>  for Python2.6 that would also need module foo, but only when
>  we actually get python2.6. Let us also have a package bar-baz that is
>  written for python2.5.

AFAICS you've correctly described the requirements for this scenario, but
this is not the scenario that I've been trying to point out to you.

The scenario where your policy draft fails to cover the needs of an
application is this:

Package python-foo packages a public pure python module.  Package bar
imports the module foo, and invokes /usr/bin/python2.3 as the interpreter.
Current is 2.4, 2.3 is still in the archive (naturally, since bar uses it),
other later versions may be available but are irrelevant; for our purposes,
assume that foo is simple enough that it's compatible with all versions of
python, past and present.

Under 4.6 of the draft python policy, python-foo does not declare any
Provides; thus:

 Package: python-foo
 Version: 1.0-1
 Depends: python

 Package: bar
 Version: 1.0-1
 Depends: python2.3, python-foo

This is the only possible expression of these package relationships which
is sanctioned by the policy draft.

Now, introduce version 2.0 of python-foo.  Because upstream considers
python 2.3 obsolete, they have begun using language features in their module
(internally, not as part of the module interface which remains unchanged)
that are specific to python 2.4 and above.

Now we have:

 Package: python-foo
 Version: 2.0-1
 Depends: python (>= 2.4)
 Provides: python2.4-foo

But this package still satisfies the dependencies of package bar, even
though bar needs python2.3 and python-foo is no longer compatible with
python2.3.  The package will be broken on partial upgrade, unless python-foo
also adds a versioned conflict on bar!  Thus, the problem arises when
support for old versions of python is dropped by a module, rather than when
new versions of python are introduced.

Versioned conflicts are not scalable, frequently confuse the packaging
system into removing packages instead of upgrading them, and will inevitably
tend to be added after the fact resulting in unnecessarily bad user
experiences in unstable (or even in testing and stable, if they escape
notice long enough).  This is why a package which needs module foo to be
available for a specific version of python, instead of for the default
version, should not be allowed to depend on python-foo.  It is the same
rationale as for requiring library package names to be changed with changes
to library sonames.

Now, whether python-foo actually declares Provides: python2.3-foo in the
first place is something I think should be left entirely up to the
maintainer; if the maintainer chooses not to declare the provides, then we
should not allow bar to be packaged, because its rightful dependencies are
"python2.3, python2.3-foo" and the latter dependency cannot be satisfied.  I
think it's perfectly ok for policy to discourage maintainers from diverging
from the default version of python -- indeed, I think it's ideal to have
only one version of python for a given stable release.  But *if* a package
is going to be in the archive which will require a non-default version of
python, I think it's important that it be packaged in a way that doesn't
require retconning with Conflicts to prevent broken package combinations on
a user's system.

> Now we have two policy proposals, A, and B. A decrees that
>  python-foo depends on python, ad has no provides. Policy B requires
>  that python-foo also provide python2.3-foo and python2.4-foo.

I don't actually think it's a good idea to require python-foo to provide
python2.3-foo and python2.4-foo.  I do think it should be *permitted*.  I
also recognize that expressing in the python policy when it is or isn't a
good idea to declare the provides would be rather complicated...


Now, looking at your example:

> The following transition events occur.

No disagreements on 1) and 2).

>  3) Python2.3 is dropped.
> policy Apolicy B
> no upload.  Upload python-foo, removing
> provides for 2.3

Why does python-foo need to drop the provides for 2.3?  In what way does
python2.3's removal from the archive mean that python-foo has stopped
supporting python2.3?

Now, if python-foo depends on an extension module and declares provides,
then yes, it would need to be reuploaded whenever python2.3-quux ceases to
be available.  This will already be true for any modules that don't support
all python versions.

Also,

>  4b) foo can't be written for version 2,6, or will take time, and
>  support for 2,6 is dr