Re: dh_python and python policy analysis
On Tue, 2006-08-08 at 13:41 +0200, Pierre Habouzit wrote: > Le mar 8 août 2006 00:18, Pierre Habouzit a écrit : > current explicitely says that the package is only built for the current > python version, and not for any other supported in debian. But I don't > like that value for the following reasons: > * it says for what the package is built, whereas other values explain >for which range of python versions the package is build-able, so >semanticaly it's not homogenous ; > * it prevents the packager to explain with which python versions the >package is compatible, as saying 'current' suggests that the package >is now compatible with the current python version, and will always in >the future, wich may be wrong when (if that happen) some python 3.0 >that is not 100% backward compatible should exists ; > * it also give an information we already have as a package that is >built for the current python version should depend upon python-dev >and not python-all-dev ; It's possible to build Python modules for all versions with only python-dev, if they are pure Python modules (feedparser is such a package, its dependency on python-all-dev is extraneous and could be just python-dev). So simply looking at the dependencies is not enough information. > * current has not a constant meaning, as it depends of the state of the >package python-defaults, and not only of the state of the archive >when the package was uploaded. This is IMHO the biggest flaw of that >field value. This is exactly the *point* of the field. It means you can just binNMU packages and support the new version. If we lose this ability we've lost much of the point of the Python transition. Packages with private extensions still cannot make use of anything but "current" to take real advantage of the new policy (things like ">= 2.3" are a lie because they can still only support one version at a time). If you get rid of it, they are back to the crappy situation we were at a year ago. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Changes to Tk 8.4 breaking Tkinter?
Hi,
I maintain the 'uligo' package, which uses Tkinter. Since recently (I'm
not sure exactly when; it worked when I uploaded it but doesn't work
now...), Tkinter has been throwing an exception when moving the mouse
over the game board or clicking:
Exception in Tkinter callback
Traceback (most recent call last):
File "/usr/lib/python2.2/lib-tk/Tkinter.py", line 1299, in __call__
args = apply(self.subst, args)
File "/usr/lib/python2.2/lib-tk/Tkinter.py", line 1032, in _substitute
e.num = getint(b)
ValueError: invalid literal for int(): ??
It seems __call__ in the CallWrapper class (which is internal to Tk) is
getting called with arguments like:
('3669', '??', '??', '??', '??', '16', '553755622', '??', '147', '407',
'??', '0', '??', '??', '.139284980.139677068.139366564', '6', '348',
'480', '??')
_substitute is then trying to convert the second argument (which I think
is supposed to be an event number) to an integer, which fails. If that
had succeeded, it would've tried to convert several other arguments to
integers, which also would've failed.
Unfortunately, my Tk-fu is nonexistent. Can anyone pinpoint where this
bug is (uligo, Tkinter, Tk 8.4?) I think it's an incompatibility between
Tkinter and Tk 8.4, but I'm not sure.
--
Joe Wreschnig <[EMAIL PROTECTED]>
signature.asc
Description: This is a digitally signed message part
Re: Python 2.3 as default?
On Sun, 2003-08-03 at 23:16, David M. Cooke wrote: > What's the word on making the default version of python to be 2.3 > instead of 2.2, now that 2.3 is released? IMO maintainers should start compiling/packaging their modules for it before any transition of the 'python' package happens. This seems to be trivial in at least some cases, where I've been using 2.3-based packages for a while (pygame, most notably. This seems to be already done in most cases, although it's not in the case that matters most to me :) (Pygame, #202805). A glance at apt-cache search's output indicates that the vast majority are ready, though. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: Python policy update
On Wed, 2003-08-20 at 07:09, Josselin Mouette wrote: > In order to make it up to date, and to match current packaging > practices, I have prepared a draft for a python policy update. > It is available at: http://people.debian.org/~joss/python/ > > It includes clarifications, a new section on bytecompilation, treats the > case of private modules, and appendix B now describes a transition like > the current one (which is likely to happen again in the future). 3.1.1: "A program using /usr/bin/python as interpreter can come up with private python modules. These modules should be installed in /usr/lib/site-python/module, /usr/lib/pythonX.Y/site-packages/module (where pythonX.Y is the current python version) or /usr/lib/package. In the latter case, this directory should be added to sys.path at the program startup. Such a package must depend on "python (>= X.Y), python (<< X.Y+1)"." pydance comes with a number of "modules", which are actually its core source code split into managable files. It installs these to /usr/share/games/pydance, and doesn't byte compile these. Based on my reading of this section, this is now not allowed anymore. However, I don't see a good reason for disallowing this. By not bytecompiling the modules, it means that the package works on any version of Python and Pygame meeting its minimum requirements, which is both 2.2 and 2.3. It is entirely platform-independent data, which means it can go into /usr/share, and be replicated across platforms more easily (although this doesn't matter much in the case of pydance specifically, I can see it mattering for other programs). Is there a reason that programs should bytecompile their modules? The startup time reduction, at least in pydance's case (6800 lines, 27 files) is unmeasurable, it takes up more space, and means backporting the package becomes necessary for testing users. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: Python policy update
On Sat, 2003-08-23 at 23:54, Donovan Baarda wrote: > On Sun, 2003-08-24 at 09:38, Joe Wreschnig wrote: > [...] > > pydance comes with a number of "modules", which are actually its core > > source code split into managable files. It installs these to > > /usr/share/games/pydance, and doesn't byte compile these. Based on my > > reading of this section, this is now not allowed anymore. > > > > However, I don't see a good reason for disallowing this. By not > > bytecompiling the modules, it means that the package works on any > > version of Python and Pygame meeting its minimum requirements, which is > > both 2.2 and 2.3. It is entirely platform-independent data, which means > > it can go into /usr/share, and be replicated across platforms more > > easily (although this doesn't matter much in the case of pydance > > specifically, I can see it mattering for other programs). > > > > Is there a reason that programs should bytecompile their modules? The > > startup time reduction, at least in pydance's case (6800 lines, 27 > > files) is unmeasurable, it takes up more space, and means backporting > > the package becomes necessary for testing users. > > The problem is, run pydance as any user with write permissions to > /usr/share/games/pydance, and the *.py's there will be compiled and > saved as *.pyc's using whatever version of python was used at the time. > When you de-install or purge pydance, the *.pyc's will be left behind. The only such user should be root, who shouldn't be running pydance anyway... > When the default python changes version, the useless *.pyc's compiled > with the old version of python will stay there until pydance is again > run by someone with write access. While the old pyc's hang around, they > take up space Space that would be used anyway if I bytecompiled them; this argument boils down to "you can use the space, or you can not use the space until the sysadmin does something stupid". The second option is definitely a better choice. > and add slightly more overhead than no pyc's at all > (python will try them first before realising they were compiled with the > wrong version). This doesn't take very long, I just tried it. Specifically, I couldn't reproduce statistically significant differences between no pycs and bad versioned pycs. Python probably embeds the version of the pyc in the file so the check is fast. And if one doesn't abuse one's administrator powers, this case never happens. OTOH, the benefits of not bytecompiling: - Unless the admin is stupid, space is saved. - The data is sharable across architectures. - Build-Depends are likely lessened. - The package migrates more easily between Python versions (and thus more easily into testing, and eventually, is more easy to pin from stable). IMO this is worth a negligable startup overhead. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: python2.3-profiler package
On Wed, 2005-02-16 at 14:12 +0100, Pavel Jurus wrote: > Are concerned developers aware of current situation? Also maybe it would > be worth trying to ask upstream authors about license because current > python distribution is broken in that sense that lots of 3rd party > software expects standard python distribution with profile.py for > example and will not work on stock debian. Packages like ipython would > have to be moved to contrib section which I think is a shame. The Python developers are aware of the problem[0], but it seems they don't care enough to do anything[1]; from Tim Peters: > Still, if that matters, perhaps Debian will need to leave these > modules out. Bold users will still be able to grab them from > any number of other places. [0] http://mail.python.org/pipermail/python-dev/2005-February/051450.html [1] http://mail.python.org/pipermail/python-dev/2005-February/051465.html -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: python's gettext.gettext broken, use gettext.lgettext
On Mon, 2005-08-08 at 07:45 +0900, Junichi Uekawa wrote: > Hi, > > While I was hacking at debconf, I noticed that > python's gettext function returns strings encoded in the > original encoding; which will appear as garbage on > the screen. The best way to do gettext in Python is to do: gettext.install(textdomain, unicode=True) Which installs ugettext as '_' function into the __builtin__ namespace. That makes _ return Python 'unicode' objects, which is what programs should be using internally anyway. This is harder if you're trying to localize a module since then you don't want to screw with __builtin__; you should use a local _ assignment instead (http://www.python.org/doc/current/lib/node329.html). It's basically what you wrote. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: python's gettext.gettext broken, use gettext.lgettext
On Mon, 2005-08-08 at 21:34 +0900, Junichi Uekawa wrote: > Hi, > > > > It is also useless for the issues at hand: since linda and > > > apt-listchanges apparently use local strings, giving them Unicode > > > strings would break them. So Junichi's change looks right to me. > > > > > Standing up for Linda, I am more than willing to fix her usage of > > gettext, and I am currently investigating using Joe's suggestion, to > > see what that gives me. > > > > Anyway, in Python, unicode string objects behave the same as normal > > string objects, so to my mind, the breakage should be minimal. > > What's broken about linda currently, and what following Joe's > suggestion will still break linda is that linda doesn't > follow the current CODESET. > > You'd expect iso-8859-1 output on stdout when the locale says so, and > utf-8 output on stdout when the locale says so. > > 'ugettext' is a python's invention of gettext which only > returns UTF-8; which you will have to call like: No, it doesn't return UTF-8, it returns unicode objects. They're automatically recoded when you try to print them (based on the same function lgettext uses, locale.getpreferredencoding()). As Steve said, unicode objects are basically like str objects, so code changes should be minimal. I'll take a look at Linda/Lintian soon to see what needs to be done, but I suspect it'll be trivial. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: downgrading optimization for m68k [was: Bug#328453: pbzip2_0.9.4-1(m68k/unstable/zeus): FTBFS on m68k]
On Fri, 2005-09-16 at 00:37 -0500, Joe Wreschnig wrote: > On Thu, 2005-09-15 at 19:47 -0700, Steve Langasek wrote: > > If you have a patch that fixes the ICEs on m68k, by all means please forward > > it to the BTS. > > > > But a larger question is, why are so many packages being built entirely with > > -O3 when policy recommends -O2? Policy does say it's ok to use other > > compiler flags if appropriate, but I'd be surprised if all of these packages > > have been benchmarked to confirm that -O3 actually gives measurable > > performance benefits. > > I don't know if it gives measurable benefits, but all Python extensions > use -O3 by default (from /usr/lib/python2.3/config/Makefile). Personally > I think it's dumb, but maybe the Python maintainers know better? This is > what triggered the bug in python-flac for me. Overriding distutils isn't > something I've figured out yet (doing so is a task for the weekend). I guess I should ask here, too. Does anyone know why Python is compiled with -O3 rather than -O2? Also, does anyone know the best way to override distutils on a per-arch basis to change that? -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: downgrading optimization for m68k [was: Bug#328453: pbzip2_0.9.4-1(m68k/unstable/zeus): FTBFS on m68k]
On Fri, 2005-09-16 at 21:00 +0200, Matthias Klose wrote: > Joe Wreschnig writes: > > I guess I should ask here, too. Does anyone know why Python is compiled > > with -O3 rather than -O2? Also, does anyone know the best way to > > override distutils on a per-arch basis to change that? > > There's only one optimization macro to build the interpreter and the > modules. IMO it makes sense to build the interpreter with -O3, and > even to build the standard modules like _sre with this optimization > level. What can be done, is to lower the opt level after compilation > in the package. > > But as you can see, even with -O2 python2.4 FTBFS on m68k. Regardless, -O3 has been historically buggy compared to -O2 on every arch. Bill Allombert mentioned it was broken on x86 right now too. And given that I've spent about 5 hours digging around trying to discover the cause of #328587 -- or even reproduce it -- with no luck, I'm about to blame -O3 for it. If there aren't numbers suggesting -O3 is a real win, I think it's a bad idea to use it *anywhere*. It's a bad default, and an especially bad one for something like Python modules that sit on top of a very complex layer of code. And that's the case regardless of whether we're in the middle of m68k breakage or not. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: How to disable optimization for Python distutils
On Sat, 2005-09-17 at 23:59 +0900, Seo Sanghyeon wrote: > Seo Sanghyeon writes: > >> To disable optimization when compiling Python modules, just set CFLAGS. > >> Example: CFLAGS=-O0 python setup.py build > > On Sat, Sep 17, 2005 at 04:24:16PM +0200, Matthias Klose wrote: > > Not correct. You suppress any other flags which are set in > > /usr/lib/python2.x/config/Makefile > > No, it doesn't. CFLAGS is *added* to flags set in Makefile. > Check Python 2.3's distutils/sysconfig.py near line 168. Awesome, thanks! -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Ending/reducing bytecode compilation, loosening dependencies
s not necessary? Without .pyc files, there's no reason for this tight dependency at all. Even if we keep .pyc files, I think loosening this requirement is a good idea. Programs will still run perfectly fine with mis-versioned .pyc files; the worst we'll see is some slightly longer startup times. How?: Strike the third paragraph from 3.1.1. This would also negate the fifth paragraph, which outlines a hypothetical overcomplicated solution to the same problem. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: Ending/reducing bytecode compilation, loosening dependencies
On Fri, 2005-12-30 at 22:59 -0600, Kenneth Pronovici wrote: > > About a month ago Steve Langasek and I discussed the state of Python > > packages on IRC, in particular the effects of bytecode compilation; the > > effectiveness (or lack thereof) of it, and how it tightens Python > > dependencies. I'd like to propose three changes to how Python modules > > are handled. > > I have a couple of questions. > > Do you intend this proposal to apply to Python libraries, or Python > applications, or both? Both. As they're at the leaves of the dependency tree, applications would benefit most from having looser dependencies. Applications are also more likely to be pure-Python, rather than having some compiled parts. Finally, applications are the points where the packager will have the best idea whether bytecompilation matters or not. > I'm thinking that many applications (especially ones like EoC) would be > built for just a single version of Python anyway. In this case, why > would it matter whether we have pre-compiled bytecode around? Well, the goal is to not "build" them at all, just have the Python source lying around. Then when the default version changes, they don't have to change in any way. > What would you suggest doing about "hybrid" packages which are primarily > applications, but also want to make their modules available to other > Python programs? Same thing. Don't bytecompile the modules. Bytecompiling is useless. pychecker is a good example, because it's a pretty straightforward example of pure-Python modules whose run time dwarfs their compile time. pychecker's packaging is also a good example of how the current policy is totally broken. Without bytecompilation the current setup is totally sane, albeit policy-violating. With bytecompilation, it's both insane (since you're storing useless bytecode) and policy-violating. > Finally, what do you suggest doing with packages that contain both > pure-Python modules and C extensions? It seems that any package which > contains a C extension is necessarily tied to a specific version of > Python, and so might as well have pre-compiled modules. You didn't seem > to address this in your proposal, but maybe that's because you're > assuming that current policy is appropriate. Well, "appropriate" might not be the right idea -- "the best I can think of for now" would be better. As long as Python changes its library/soname each version, there's not much we can do about this. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: Ending/reducing bytecode compilation, loosening dependencies
On Sun, 2006-01-01 at 19:09 +0100, Josselin Mouette wrote: > > 2. Stop compiling .pyc files (this I expect to be contentious), unless a > > package wants to. > > > > Rationale: .pyc files have a minimal gain, and numerous failings. > > > > Advantages of .pyc files: > > * .pyc files make Python imports go marginally faster. However, > >for nontrivial Python programs, the import time is dwarfed > >by other startup code. Some quick benchmarks show about 20% gains > >for importing a .pyc over a .py. But even then, the wall-clock time > >is on the order of 0.5 seconds. Lars Wirzenius mentioned that > >this time matters for enemies-of-carlotta, and it probably also > >matters for some CGI scripts. > > Do you have more complete figures about this time? Getting figures for this is difficult, because it varies a lot based on the length/complexity of the code, and the differences are just long enough that they matter for some things but not others. I'll try to put together a proper benchmark over the next few days. But IMO this is backwards; it should be the responsibility of those who want to optimize to show that it matters. > > * Generating them at compile-time means they won't accidentally > > get generated some other time. > > > * .pyc files are not really architecture independent. > > Which is why pure python modules are still in /usr/lib. Well, no, pure Python modules are in /usr/lib and /usr/share. Python policy says they should go in /usr/lib, but that's irrelevant given how many packages ignore it (dput, gajim, pydance, gdesklets, hal-device-manager, ipodder, linda, pygtk, quodlibet, reportbug, rss2email on my system). > > * .pyc files result in strange bugs if they are not cleaned up > >properly, since Python will import them regardless of whether > >or not an equivalent .py is present. > > Which makes things worse if they aren't cleaned correctly. > > The real fix for this issue isn't to stop generating the .pyc files. It > is to make python not generate any .pyc files at all when running as > root. Well, I would say, not generate any pyc files unless requested, regardless of whether or not the user is root. My goal was to propose this with minimal changes to the Python code, but I'm also open to doing this, of course. > > 3. Python dependencies should be loosened (and here I expect a > > flamewar). > > ... > > We could indeed remove the requirement from the python policy, but I > don't like the idea of implementing any solution that makes > outdated .pyc files lie around on the system. > > I can see two ways to go from now for dh_python. > > 1. Make, as you suggested, dh_python stop building .pyc files - but only > for python-only packages, by default. The former behaviour could be > enabled by a flag for modules that are affected enough by the > performance hit. For this solution to work, it would be better to make > python stop creating .pyc files when run as root. > > 2. Make a separate python-support (inspired from emacsen-support) > package that takes care of the byte-compilation. This is the solution > I've been working on the last weeks, but it isn't ready yet. In my opinion, 1 is the better course. As you said, 2 is a lot of work. Since I'm not convinced bytecode is desirable in the general case, I don't think it's worth maintaining such a python-support rather than just not using bytecode. I would be interested in seeing what you have so far. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: Ending/reducing bytecode compilation, loosening dependencies
On Mon, 2006-01-02 at 18:31 +0100, "Martin v. Löwis" wrote: > Joe Wreschnig wrote: > > 1. Stop compiling .pyo files, entirely (I'm hoping for little > > argument on this). > > I agree. > > > How?: compileall.py:57, -cfile = fullname + > > (__debug__ and 'c' or 'o') +cfile = fullname + 'c' > > This is the wrong solution, though. If Python isn't invoked > with -O, the extension will always be 'c'. The right solution > is to invoke compileall only once, without -O. > > If you apply the proposed change, the .pyc files will be generated > twice (once for without -O, and once for with -O), you will end > up with the .pyo file named .pyc. D'oh, thanks. Then the correct place to patch this is in dh_python's postinst generator; it's still trivial, though. > > Disadvantages: * They waste disk space; they use about as much as the > > code itself. > > That's not true (or, rather, I disagree): they don't "waste" the > disk space, and the code is not "about" the same as the source code > size. Rather, the size varies: for Python 2.4, htmlentitydefs.py > is 18054 bytes, and htmlentitydefs.pyc is 6405 bytes (which is > roughly a third). > > > * It's still far too easy for modules to be regenerated for the wrong > > version of Python; just run the program as root. > > I don't understand. What precisely would root do, what precisely > would happen, and why would that be a problem? > > Currently, modules are installed with one for each Python > version. So if root runs a program, the Python interpreter picks > up the modules of that Python version, for which .pyc files are > already generated, so nothing is regenerated when root runs > a program. This is not the case; many Python modules are in /usr/share/program, /usr/lib/program, or an otherwise unversioned directory. > Even if you have a package which is installed just in a shared > way (e.g. in site-python, not site-packages), root would > generate .pyc files for some Python version, yes. Why would > that be a problem? If generating the wrong pyc files is not a problem, why do we care about generating them at all? Either 1) .pyc files are important. Then we should be sure we're compiling then right. This doesn't happen under the current system. 2) .pyc files aren't important. Then it doesn't matter if we compile them at all. But not compiling them is not allowed with the current system. The current system goes against both sane conclusions; I advocate 2 (because I think it's easier, and I don't think pyc files matter), and Josselin Mouette has made some excellent progress towards 1. > Even if generating bytecode files during installation is > suppressed, why would that make it harder for root to > regenerate the modules for the wrong version? > Why (Did you lose a sentence here?) As I said to Josselin, one goal with the proposal was minimal changes to Python's code initially, so discussion could be around the usefulness of bytecode rather than the details of a particular patch. I have no problem patching Python to not generate modules at all unless requested. > > > * .pyc files are not really architecture independent. > > Right. Fortunately, they are created installation, instead of being > shipped in the debian package. As you say, they are > platform-dependent only in border-cases. But they're also placed in /usr/share, which can be shared across many systems, so this doesn't help. > > Another note: Currently, Python policy is based around the assumption > > that .pyc files are valid within a single minor Python revision. I > > don't find any evidence to support this in the Python documentation. > > Nevertheless, this is a promised guarantee. If you wish it documented, > just file a bug report. Okay. http://sourceforge.net/tracker/index.php?func=detail&aid=1395511&group_id=5470&atid=105470 > > How?: dh_python should not call compileall.py unless give some > > special flag. Python policy 2.5 should change "should be generated" > > to "may be generated." > > What's the point in having a policy, then? The policy should either > mandate it (and specify additional conditions if it's not an > all-or-nothing deal), or forbid it. Why? If the speed matters in some case, by all means use it! But don't do it by default, because of the disadvantages of pyc files. This is similar to how Debian policy suggests -O2 by default, but allows programs that do benefit from more aggressive optimizations to use them, despite the disadvantages in doing so. > > 3. Py
Re: python packaging infrastructure
var/lib/python-support/bytecompiled. I've built several packages with private Python libraries to test it, and it keeps a list of those directories and rebuilds them. I didn't even have to specify them manually. update-python-modules -b -a is recompiling them. > - python-support's dh_python currently does more on it's own than > dh_pycentral does. > I'm in favour of supporting an infrastructure handling extensions and > modules outside the default sys.path well, which ever this > implementation will look like. It seems to me that it would be easier to extend python-support, which is already in the archive (and as far as I can tell, working basically as intended) to use the proposed control field. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: python packaging infrastructure
On Fri, 2006-01-13 at 00:33 -0600, Joe Wreschnig wrote: > Unless I misunderstand (either or or python-support) it does, Should be ^- "either you or python-support" -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: python packaging infrastructure
On Fri, 2006-01-13 at 11:05 +0100, Josselin Mouette wrote: > Le vendredi 13 janvier 2006 à 00:33 -0600, Joe Wreschnig a écrit : > > > - Some concerns were raised by the release team, that python-support > > > tracks it's own state instead of using the dpkg database. To keep > > > track of usable python versions, python-central uses a field > > > Python-Version in the control file (having values like "2.3, 2.4", > > > "all", ">= 2.3") > > > > Is there some reason python-support couldn't be extended to also do > > this? I agree it is a better solution than a separate database. > > I intend to make python-support use a .pyvers file listing the supported > versions. This file would be generated by dh_python, when using > dh_python -m 2.2 (for just a minimal version) or copied from > debian/$package.pyvers. > > Having a new field in the control file diverges from how debhelper > programs usually behave, so I'd like to avoid such a solution - and I > don't think there's a need for it. Well, there is a need for a database of some kind. And I think given the nature of the information, dpkg seems a good place to keep it. Note that we're not talking about "debhelper programs" here, but all Python packages. Python policy shouldn't mandate debhelper, even if debhelper can enable packages to conform with the policy. As a maintainer, I wouldn't find it confusing or inconvenient to have another field in the control file. In fact, it would probably make it easier to update, since in what you propose Python version handling would be split between debian/control and debian/rules (as it is now); in Matthias's system, it would all be in debian/control. Which makes sense, because it's install-time package relationships, not build instructions. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: Ending/reducing bytecode compilation, loosening dependencies
On Fri, 2005-12-30 at 12:33 -0600, Joe Wreschnig wrote: > 1. Stop compiling .pyo files, entirely (I'm hoping for little argument > on this). > > Rationale: .pyo files are a joke. They aren't optimized in any > meaningful sense, they just have asserts removed. Examples for several > non-trivial files: Although I stand by this, it was pointed out on IRC that Python won't load pyc files when given -O, only pyo. So python -O needs to recompile everything if you don't have pyo files. Personally I consider anything running python -O a bug waiting to happen, but that isn't a universally-shared opinion. The new systems being discussed should hopefully make it easier to transition to optional pyo compilation (and I would hope, default to off). -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: python packaging infrastructure
On Fri, 2006-01-13 at 12:04 +0100, Matthias Klose wrote: > Joe Wreschnig writes: > > On Thu, 2006-01-12 at 14:44 +0100, Matthias Klose wrote: > > >- modules are installed into a directory not directly in sys.path. > > > > While I understand the rationale here, this could be *very* confusing > > for people coming to Debian from other operating systems. > > People will find their modules in the standard place after > installation, handle by a script in the postinst. The only place > where you do see the difference will be the output from some tools > like `dpkg -L' and `dlocate'. Indeed; what I meant was that we should make sure this is very well-documented, not just for the packagers, but for any Python developers who might be surprised as well. One issue that comes to my mind now is behavior with regards to dpkg-divert. I've diverted a number of Python modules on my system; under this centralized registry I would have to divert them for all versions at once (which is fine for me now, but maybe not everyone), or better, make python-whateveritscalled understand diversions of the target (per-version) module names. > Private modules should be kept as they are. The existance of a > Python-Version attribute lets the central/support package know, that a > byte compilation with the new version is needed. Great, this is exactly what I was hoping for. > > Is this really feasible? The installed size will be, rough, source size > > * (the number of Python versions you have installed + 1), then * 2 if > > you have pyo files as well, since we're compiling for multiple versions. > > Since the installed size would vary based on the number of Python > > versions you have installed, an accurate report via APT would be > > impossible. > > dpkg does have support for the installed size, you could do that; > unless it's not done by some helper script nobody will use it anyway ;) Yes, but afaik Installed-Size is mostly used by APT to gauge disk space deltas when upgrading. Since it wouldn't know until the package was unpacked (and how many copies of the modules needed to be compiled were figured), this would prevent the most common case from working no matter what. It also wouldn't know that installing a new Python version would cause a lot of disk space usage as all the new modules were recompiled. So I think it's best not to guess at all, and at least be consistently inaccurate. > > This also blocks any Python program using dh_python from reaching > > testing until the issue is resolved, so it behooves us to do this > > quickly. > > that's understood. it's unfortunate that the package wasn't uploaded > to experimental first to sort out things. otoh, the change to > dh_python could be reverted in debhelper without doing any harm. I would also be very happy with reverting the dh_python changes, since right now, any built Python package is probably going to be completely useless regardless of what we end up doing. > > > Some things about python-support, others may comment about python-central: > > > > > > - Some concerns were raised by the release team, that python-support > > > tracks it's own state instead of using the dpkg database. To keep > > > track of usable python versions, python-central uses a field > > > Python-Version in the control file (having values like "2.3, 2.4", > > > "all", ">= 2.3") > > > > Is there some reason python-support couldn't be extended to also do > > this? I agree it is a better solution than a separate database. > > Joss didn't yet comment. I really do not want to have a packager/user to > understand two different things for extensions and modules, and > probably a third one for private modules. I guess I still don't totally understand -- from what I can see, python-support handles public and private modules already, and doesn't (can't, really) need to be invoked for extensions. I'm not sure any tool will significantly help C extensions, but that's okay, because I also don't see much wrong with how they're handled right now. The only potential change being storing all binary versions inside one .deb, which I can go either way on. Since the only extension module I'm going to maintain by the end of next week is huge and need separate versions anyway, I'd rather that be decided by the people it affects. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: python packaging infrastructure
On Fri, 2006-01-13 at 12:44 +0100, Matthias Klose wrote: > Joe Wreschnig writes: > > One issue that comes to my mind now is behavior with regards to > > dpkg-divert. I've diverted a number of Python modules on my system; > > under this centralized registry I would have to divert them for all > > versions at once (which is fine for me now, but maybe not everyone), or > > better, make python-whateveritscalled understand diversions of the > > target (per-version) module names. > > By diverting the directory name which is not in sys.path, you could > divert all versions at once, diverting the directory in > /usr/lib/pythonX.Y you can divert them by version. > > Ok, I think that something else, i.e. how to handle different versions > of a module. Looking at the benefits(?) of eggs ;). What about the > following setup: /usr/lib/pythonX.Y/dist-packages is added to > sys.path, python policy requires to install distribution packages into > this place, maybe supported by a distutils patch adding a new option > to setup.py. when you do build your own modules these are installed > into /usr/lib/pythonX.Y/site-packages. By changing the order of the > two directories you can priorize your locally installed modules. (yes, > we do have the /usr/local/lib/pythonX.Y/site-packages directory as > well, but making this the default would break the existing packaging). This sounds like an excellent solution. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: when and why did python(-minimal) become essential?
On Thu, 2006-01-19 at 12:12 +1000, Anthony Towns wrote: > debian-python Cc'ed > > On Wed, Jan 18, 2006 at 07:02:32PM -0600, Joe Wreschnig wrote: > > > This is something that Python upstream explicitly does not want; the only > > > reason for creating python-minimal was so that it could be Essential: yes, > > > not to support stripped-down Python installations. > > So why does Debian need/want python-minimal? > > (This is a question mostly for Matthias, I think, but if you know the > > answer that's great.) > > Some reasons: > > * compatability with Ubuntu -- so that packages can be easily ported back > and forth between us and them; I expect most of the work ubuntu might do > on improving boot up will require python-minimal This would be nice. Right now it's accomplished through patches Ubuntu makes to dh_python and cdbs. They'd probably like to drop those. > * allowing us to easily use python (as well as C, C++ and perl) for programs > in the base system I wouldn't mind this, but it does seem to be somewhat against the definition of "base." > * allowing us to provide python early on installs to make users happier This feels weak to me; it applies equally well to any language a user might want. > I don't know what's actually in (or more importantly not in) > python2.4-minimal though. I'm eyeballing right now. Things that jump out at me: * No character encoding, translation, or locale handling. * A little oddly, loss of shutil. * No sockets. The first one seems like it would be a show-stopper to me, unless we expect programs in the base system to only deal with ASCII. This is a fairly large addition to package, too. The second can easily be fixed; possibly just oversight. It's a small module and gives Python equivalents of cp -r, rm -r, and mv. The third seems like something software in base may want to do; I mention it specifically because perl-base include socket support. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: when and why did python(-minimal) become essential?
On Thu, 2006-01-19 at 09:31 +, Colin Watson wrote: > On Wed, Jan 18, 2006 at 11:36:13PM -0600, Joe Wreschnig wrote: > > On Thu, 2006-01-19 at 12:12 +1000, Anthony Towns wrote: > > > Some reasons: > > > > > > * compatability with Ubuntu -- so that packages can be easily ported > > > back > > > and forth between us and them; I expect most of the work ubuntu might > > > do > > > on improving boot up will require python-minimal > > > > This would be nice. Right now it's accomplished through patches Ubuntu > > makes to dh_python and cdbs. They'd probably like to drop those. > > As a point of information, Ubuntu doesn't patch dh_python at present, > and I don't see any Python-related changes in cdbs at the moment either. Oh, hrm. So packages that need to use python-minimal manually handle their Python dependencies? That seems like a significant step backwards, in terms of handling transitions. > $ dpkg -c > /mirror/ubuntu/pool/main/p/python2.4/python2.4-minimal_2.4.2-1ubuntu2_i386.deb > | grep socket > -rw-r--r-- root/root 49608 2006-01-17 12:59:02 > ./usr/lib/python2.4/lib-dynload/_socket.so > -rw-r--r-- root/root 12876 2006-01-17 12:58:18 > ./usr/lib/python2.4/socket.py D'oh. Apparently I'm blind. Thanks. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: when and why did python(-minimal) become essential?
On Sat, 2006-01-21 at 01:48 -0800, Thomas Bushnell BSG wrote: > Matt Zimmerman <[EMAIL PROTECTED]> writes: > > > One example is .config maintainer scripts, some of which are quite complex > > and worth writing in a higher-level language than shell. > > This is surely true; Steve Langasek asked if this was a real issue in > Ubuntu or merely a potential issue. > > Granted if it is a real issue, then why not use perl? Yes, I hate > perl too, but really, the argument "hey, people like Python too" > implies that we should have a scheme interpreter, a perl, a python, > emacs lisp, and well, everything anyone might want. > > Or, we say "we aren't going to support *every* high-level language" > and stick to one. There's nothing that prevents us saying "we aren't going to support every high-level language" and stick to more than one (we already stick to two -- sh and Perl). It just means "I'd like to write scripts in X" alone isn't a good enough reason. Python is the "official" language of Ubuntu. If we want to merge work they're doing (Anthony Towns mentioned their work on boot speed, for example) it's a good idea to structure our Python like theirs is. This seems to be a good reason to consider python-minimal and some form of Python in Essential. The real issue here is that the original upload didn't do that; it went through the motions without actually changing our Python packaging or upgrading the version, so we just got all of Python as Essential. No one wanted that. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: when and why did python(-minimal) become essential?
Don't reply to me directly. I should not have to tell you this. On Sat, 2006-01-21 at 13:03 -0800, Thomas Bushnell BSG wrote: > > Python is the "official" language of Ubuntu. If we want to merge work > > they're doing (Anthony Towns mentioned their work on boot speed, for > > example) it's a good idea to structure our Python like theirs is. This > > seems to be a good reason to consider python-minimal and some form of > > Python in Essential. > > This does not scale. If each Debian derivative chooses a different > "official language", and we put each of them in Essential, then we end > up with every language in Essential. We can burn those bridges when we come to them. Right now there's only one such distribution, with one such language, which has already done all the work to strip it down to a small size. Unless you expect some derived Debian distribution to use Scheme some day, this is sophistry. If you really do expect that, it's insanity. > What I hear is *not* that Python is the official language instead of > Perl, but that it is the official language *in addition to* Perl. So > now, why? Remember, "I'd like to write scripts in X" is not a good > enough reason, so what is the reason for having two official > languages? I don't manage Ubuntu policy, nor do I want to. I am a Debian developer interested in Debian. The argument for Debian is not "I'd like to write scripts in X" but "There is this large body of people writing scripts in X, and it'd be nice if we could work with them." -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
pylibs and pydeps: shlibs-like support for Python extensions
The immediate problem: PyGTK encourages other Python modules to use its
code generator and GObject wrappers. Between 2.6 and 2.8 the ABI changed
so that extensions build against 2.8 do not work against 2.6. This is
breaking gst-python and its reverse dependencies, soundconverter,
quodlibet, and istanbul. It may break other packages as well.
The general problem: Python modules that provide build-time shared
objects have no way of expressing dependencies like this. PyGTK is the
first example to break, probably because it's very large and widely
used. However, it's not unique in this respect: Pygame uses shared
objects from Numeric, for example.
The solution: Debian solved this problem for regular shared objects with
shlibs files. So the attached changes to dh_python and Python policy are
an implementation of something like shlibs files, for Python.
Open questions:
* Right now, the pylibs files are stored in /var/lib/dpkg/info. If
the dpkg maintainers don't like this, they can easily be moved
elsewhere, but this seemed like a good place given their analogous
purpose to shlibs files.
* I've not written any debhelper code before, and so my patch might
have a dumb mistake in it. My Perl is also rusty.
* See the note on package aliases in the policy update.
--
Joe Wreschnig <[EMAIL PROTECTED]>
--- /usr/bin/dh_python 2006-01-16 16:36:44.0 -0600
+++ dh_python 2006-01-31 16:03:41.0 -0600
@@ -18,7 +18,11 @@
dh_python is a debhelper program that is responsible for generating the
${python:Depends} substitutions and adding them to substvars files. It
-will also add a postinst and a prerm script if required.
+will also add a postinst and a prerm script if required, and install a
+pylibs file if one is present.
+
+If you are using a pydeps module list, it will also fill ${python:Depends}
+with the appropriate versioned dependencies from their pylibs files.
The program will look at python scripts and modules in your package, and
will use this information to generate a dependency on python, with the
@@ -109,6 +113,11 @@
s#^/##;
}
+# pylibs support
+# FIXME: This may not be the best place; it also requires approval
+# from the dpkg developers.
+use constant PYLIBS_DIR => '/var/lib/dpkg/info';
+
# dependency types
use constant PROGRAM => 1;
use constant PY_MODULE => 2;
@@ -210,17 +219,54 @@
}
}
+ # Install a pylibs file, if we have one.
+ if (my $pylibs_file = pkgfile($package, "pylibs")) {
+ doit("install", "-D", $pylibs_file,
+ tmpdir($package) . "/DEBIAN/pylibs");
+ }
+
+ # Find potential pylibs dependencies
+ my %pylibs_modules = ();
+ if (my $pydeps_filename = pkgfile($package, "pydeps")) {
+ open PYDEPS_LIST, $pydeps_filename;
+ foreach my $pylibs_pkg () {
+ chomp $pylibs_pkg;
+ my $pylibs_file = PYLIBS_DIR . "/$pylibs_pkg.pylibs";
+ if (-f $pylibs_file) {
+open(PYLIBS_FILE, $pylibs_file);
+my @pylibs_deps = ;
+chomp @pylibs_deps;
+$pylibs_modules{$pylibs_pkg} = [EMAIL PROTECTED];
+ }
+ }
+ }
+
# Dependencies on current python
$dep_on_python = 1 if $deps;
$strong_dep = 1 if($deps & (PY_MODULE|SO_MODULE));
if ($dep_on_python) {
addsubstvar($package, "python:Depends", $python, ">= $python_version");
+
if ($strong_dep) {
addsubstvar($package, "python:Depends", $python, "<< $python_nextversion");
} else {
addsubstvar($package, "python:Depends", $python, "<< $python_nextmajor");
}
+
+ # pylibs dependencies are meaningless without a shared
+ # object module, but if they've included a pydeps file,
+ # use them anyway.
+ for my $pylibs_pkg (keys %pylibs_modules) {
+ my $r_pylibs_versions = $pylibs_modules{$pylibs_pkg};
+ my @pylibs_versions = @$r_pylibs_versions;
+ foreach my $pylibs_ver (@pylibs_versions) {
+if ($pylibs_ver) {
+ addsubstvar($package, "python:Depends",
+ $pylibs_pkg, $pylibs_ver);
+}
+ }
+ }
}
my $need_prerm = 0;
3.3. Extensions Used By Other Extensions
If your package contains compiled Python extensions that other
extensions use, or otherwise creates a versioned dependency at
build-time, it should provide pylibs files. A pylibs file is similar
to an shlibs file; it provides version dependency information for
other packages at build time.
The pylibs file format is one version clause per line, e.g. the
contents of /var/lib/dpkg/info/python-gtk2.pylibs might be:
>= 2.8
<< 2.10
If your package builds extensions that depend on other binary
extensions, or has dependencies otherwise affected by the versions of
installed modules at build-time, you should use a pydeps file. If you
are using dh_python, you can make one named debian/pydeps or
debian/.pydeps and it will be automatically
Re: pylibs and pydeps: shlibs-like support for Python extensions
On Wed, 2006-02-01 at 16:16 +0100, Josselin Mouette wrote: > Le mardi 31 janvier 2006 à 16:08 -0600, Joe Wreschnig a écrit : > > The immediate problem: PyGTK encourages other Python modules to use its > > code generator and GObject wrappers. Between 2.6 and 2.8 the ABI changed > > so that extensions build against 2.8 do not work against 2.6. This is > > breaking gst-python and its reverse dependencies, soundconverter, > > quodlibet, and istanbul. It may break other packages as well. > > > > The general problem: Python modules that provide build-time shared > > objects have no way of expressing dependencies like this. PyGTK is the > > first example to break, probably because it's very large and widely > > used. However, it's not unique in this respect: Pygame uses shared > > objects from Numeric, for example. > > > > The solution: Debian solved this problem for regular shared objects with > > shlibs files. So the attached changes to dh_python and Python policy are > > an implementation of something like shlibs files, for Python. > > I don't think the proposed solution addresses the specific issue you > describe (PyGTK ABI breakage). > > First, when a library package breaks the way you describe, the SONAME is > changed, and the package name is changed accordingly. Same problem, same > solution: wouldn't it be better to simply change the python-gtk2 package > name? That causes unneeded transitions for programs/modules that depend only on the Python-exposed API (which is most of them). > Second, the shlibs mechanism was introduced because it is easy to obtain > (through objdump) a list of dependencies for a binary. Your patch only > seems to read the .pydeps file, which still has to be written by the > maintainer. It then relies on the .pylibs of the package you depend > upon, which has to guess when the ABI can break in the future. Well, not "guess". The maintainer should ask upstream when stuff will break. In the case of PyGTK, upstream was certainly aware of the issue. They just didn't care. You're right that it's suboptimal, but it's far more managable than the existing system where we cross our fingers and hope things don't break. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: pylibs and pydeps: shlibs-like support for Python extensions
On Wed, 2006-02-01 at 16:16 +0100, Josselin Mouette wrote: > Le mardi 31 janvier 2006 à 16:08 -0600, Joe Wreschnig a écrit : > > The immediate problem: PyGTK encourages other Python modules to use its > > code generator and GObject wrappers. Between 2.6 and 2.8 the ABI changed > > so that extensions build against 2.8 do not work against 2.6. This is > > breaking gst-python and its reverse dependencies, soundconverter, > > quodlibet, and istanbul. It may break other packages as well. > > > > The general problem: Python modules that provide build-time shared > > objects have no way of expressing dependencies like this. PyGTK is the > > first example to break, probably because it's very large and widely > > used. However, it's not unique in this respect: Pygame uses shared > > objects from Numeric, for example. > > > > The solution: Debian solved this problem for regular shared objects with > > shlibs files. So the attached changes to dh_python and Python policy are > > an implementation of something like shlibs files, for Python. > > I don't think the proposed solution addresses the specific issue you > describe (PyGTK ABI breakage). > > First, when a library package breaks the way you describe, the SONAME is > changed, and the package name is changed accordingly. Same problem, same > solution: wouldn't it be better to simply change the python-gtk2 package > name? Sorry, besides my other response, I need to answer: No, the soname stays the same. When things are upwards-compatible, we bump shlibs. We don't change the package name. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: some issues with the proposals for the python packaging infrastructure
On Thu, 2006-02-02 at 21:09 +0100, Matthias Klose wrote: > Josselin Mouette writes: > > There is still a situation we can improve easily, though: private > > modules. Currently, they have to migrate with python transitions, and > > this is only because of byte-compilation. The python-support way of > > doing things should still be fine for them, and it can reduce the number > > of packages to migrate at once, without complicating anyone's work. > > that is, packages with private modules but without extension modules > and no modules in /usr/lib/python2.x. how many packages are this? Off the top of my head and in no particular order: pydance, solarwolf, pathological (this is standard practice in the Pygame community), uligo, linda, pychecker, amarok, reportbug, dput, python-gtk2-dev, straw, gdesklets-data, hal-device-manager. My brain isn't an exhaustive list of Python packaging irregularities though, so I'm going to guess there's more. It's hard to find these automatically because there's no consistent way they're packaged. amaroK, for example, doesn't byte-compile the modules at all. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: some issues with the proposals for the python packaging infrastructure
On Thu, 2006-02-02 at 23:02 +0100, Josselin Mouette wrote: > Le jeudi 02 février 2006 à 21:09 +0100, Matthias Klose a écrit : > > Josselin Mouette writes: > > > There is still a situation we can improve easily, though: private > > > modules. Currently, they have to migrate with python transitions, and > > > this is only because of byte-compilation. The python-support way of > > > doing things should still be fine for them, and it can reduce the number > > > of packages to migrate at once, without complicating anyone's work. > > > > that is, packages with private modules but without extension modules > > and no modules in /usr/lib/python2.x. how many packages are this? > > A first estimation by grepping through Contents-i386.gz gives 297 > packages. It certainly includes a lot of false positives, but gives the > order of magnitude. This sounds far too high. Are you including packages with private .py files that are not modules? This is common, and one of the reasons I said it was hard to gauge. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: [EMAIL PROTECTED]: Re: Python packaging questions]
On Sat, 2006-02-04 at 10:13 +0900, Sanghyeon Seo wrote: > But in case of feedparser and web.py, aren't they just a single file? > I can't blame upstream for not interested in setup.py. I'm not sure > what Joe Wreschnig means when he says it's "unmanagable". I think > simple dh_install is fine for a single file package. I say this for the same reason that we use autoconf instead of a shell script full of cps. It seems like overkill, but in the long run it's the only thing that has the flexibility you need. If you're doing the copying manually, it means you need to tweak the Debian install scripts for any kind of location change. If you're using setup.py, it will automatically follow the local Python settings -- whether that's Debian, Ubuntu, or a local source install. If upstream is making a Python module, that is intended to be installed system-wide, I can't imagine why they don't want a setup.py. It's like not wanting a makefile. If upstream is making a Python module that they expect to be included inside the source of other packages directly (which is not unheard of -- feedparser is often used this way), then you need to clue them in about the problems with that (security updates, bug fixes, wasted disk space). Even if they insist, that's not acceptable for Debian. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Wither the Python 2.4 migration?
Hi Matthias, What's the status of the Python 2.4 transition? During January you said you were waiting on feedback from Steve Langasek and Josselin Mouette, but Steve said he hasn't heard anything from you in a while, and thinks that the transition outweighs whatever other Python improvements you're working on. A couple weeks ago you told me (on IRC) that you'd have a new python-central/support-like thing ready for me to look at in a few days. I still haven't heard anything from you. I have to agree with Steve, if this is holding up the transition, it should stop -- the focus of the Python packages right now should be the 2.4 transition. From the looks of the Python buglists, and your responses on bugs like #340791, it looks like you might not have enough time to maintain Python anymore. Have you considered group or co-maintainership? Facing a transition may not be the best time to start it, but this needs to start very soon. The etch release plan starts calling for freezes by July, and I know you're going to need time working on other parts of the toolchain before then. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: New python policy
On Fri, 2006-06-02 at 12:52 +0200, Raphael Hertzog wrote: > * the dependencies (hopefully created automatically by dh_python) will > indicate the right interval automatically: > right now for example it would be "python (>=2.3), python (<< 2.5)" > for a package saying "XC-Python-Version: all" What's the point of such a tight upper dependency? The whole idea was to reduce the work for transitions, this still requires a rebuild when we upgrade to Python 2.5 (or in fact, whenever the supported versions of Python change). Such information is necessary for extensions, but irrelevant for programs or other modules. > * the packages should provide all "python2.X-foo" > that they really provide (i.e. it must of course correspond to the > content of the package, which in turn should correspond to what > XC-Python-Version says). This Provides field can also be generated > by dh_python. (This is particularly relevant for binary modules of > course but it can make sense for non-binary modules as well, for example > if they don't support an old version anymore, stopping to provide > python2.3-foo would for example forbid upgrading a /usr/bin/python2.3 > application relying on it) > > For applications: > * if they use /usr/bin/python, they should simply depend on python-foo > modules that they use. > * if they use /usr/bin/python2.X, they should depend on python2.X-foo > modules that they use. IMO use of python2.x-foo should be discouraged. Maintainers of applications that need specific Python versions should ask maintainers of the modules they need, and only the ones they need, to provide the version-specific virtual package. > For private modules (not sure here): > * no change except that they can also use XC-Python-Version to get > a good python dependency automatically See my first comment. This misses the point, if an application has to be rebuilt whenever the Python version changes. Then all we get is a precarious tower of tools to do exactly what we did before. Given that we still have a ton of (virtual) python2.x-foo packages under this policy, and any Python package would still need to be rebuilt during a transition, I don't see how this policy does anything useful. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: New python policy
On Fri, 2006-06-02 at 14:43 -0700, Steve Langasek wrote: > On Fri, Jun 02, 2006 at 11:50:53AM -0500, Joe Wreschnig wrote: > > On Fri, 2006-06-02 at 12:52 +0200, Raphael Hertzog wrote: > > > * the dependencies (hopefully created automatically by dh_python) will > > > indicate the right interval automatically: > > > right now for example it would be "python (>=2.3), python (<< 2.5)" > > > for a package saying "XC-Python-Version: all" > > > What's the point of such a tight upper dependency? The whole idea was to > > reduce the work for transitions, this still requires a rebuild when we > > upgrade to Python 2.5 (or in fact, whenever the supported versions of > > Python change). > > > Such information is necessary for extensions, but irrelevant for > > programs or other modules. > > Indeed, the intent was that this strict upper bound would apply to > extensions, not to modules. Not sure if that intent got lost somewhere > along the way. > > > IMO use of python2.x-foo should be discouraged. Maintainers of > > applications that need specific Python versions should ask maintainers > > of the modules they need, and only the ones they need, to provide the > > version-specific virtual package. > > Yes, this was also discussed in the BoF, with the same conclusion: because > providing python2.x-foo can only be done safely if the package depends on > the python2.x versions of all other modules it requires, making transitions > more brittle as a result, these virtual packages should really only be > provided in response to specific demand for them. For those of us who were not at the BoF and only getting sketchy information about it (e.g. Raphael's information is wrong, or at least horribly incomplete, I guess, and presentation slides are not a good summary of technical policy), when are we going to have a real proposal to comment on, or at least follow? What happened to "Python 2.4 first, new infrastructure later"? -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: python-central vs python-support
On Sun, 2006-06-04 at 20:56 +0200, Raphael Hertzog wrote: > Hello, > > now you know a bit better the policy (or at least the generic idea, feel > free to discuss details further), No. After the previous thread I am still in the dark on: - Tight upper dependencies. We you just incorrect, or are they actually required? - python2.x-* packages -- are they needed? desirable? Steve and Matthias gave different answers, and if they're present migrations end up just as fragile as they are now. Until these two issues are clear, everything else is irrelevant. The goal shouldn't be to pick a tool, but to solve the problem. If neither tool solves the problem, or policy prevents us from using it to solve the problem, then this is just technical sophistry. > 2/ Extensions can't be shared between several python versions so they need to > be compiled once for each. The packaging needs to be modified to do those > compilations. We really need a tool (maybe dh_python with a special flag) > to generate dynamically the list of python version that the package must > be compiled with. The .so files must be installed in /usr/lib/python2.X/ > and the associated .py files may be moved to a shared directory (either > the python-support or python-central directory). Was there consensus about whether or not extensions for all versions should be included in one binary package? This was not mentioned in your policy email. I'm ambivalent on the issue, I guess, but we should choose one way or the other. > I agree with Matthias that it would be better to use only one "helper > tool" but I would favor python-support instead of python-central for the > reasons outlined below. Beware this is only *my* opinion. I'll gladly > follow the consensus and would even accept to let both tools coexist. > > python-support: > * 247 lines of code > * maintained by Josselin Mouette (dh_python author) > * positive aspects > * already in the archive, works, well tested This IMO is the most important part. It's here, and we're using it. python-central has "right arond the corner" for months. Many modules are already using -support, and it integrates very easily with any project using distutils. With proper debhelper integration it would be even simpler. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: python-central vs python-support
On Sun, 2006-06-04 at 23:05 +0200, Raphael Hertzog wrote: > On Sun, 04 Jun 2006, Joe Wreschnig wrote: > > On Sun, 2006-06-04 at 20:56 +0200, Raphael Hertzog wrote: > > > Hello, > > > > > > now you know a bit better the policy (or at least the generic idea, feel > > > free to discuss details further), > > > > No. After the previous thread I am still in the dark on: > > - Tight upper dependencies. We you just incorrect, or are they > >actually required? > > For extensions they are required. For arch:all modules they're not > required since python-support / python-central can make them instantly > available to a new python version. > > > - python2.x-* packages -- are they needed? desirable? > >Steve and Matthias gave different answers, and if they're present > >migrations end up just as fragile as they are now. > > There's only python2.X-* _virtual_ packages. They are desirable but not > needed unless a python application which doesn't use the default python > version > requires a specific python-version of a module. > > However I don't see the problem of providing systematically python2.X-foo > (in particular for extensions). > > > Until these two issues are clear, everything else is irrelevant. The > > goal shouldn't be to pick a tool, but to solve the problem. If neither > > Which problem? The problem that Python migrations are currently very fragile due to strict dependencies and interconnectedness of many modules. I'm finally beginning to piece together the conclusions of the BoF, but there were many missing details (and any other onlookers are probably even more confused than me). It's very difficult to discuss this without a clear statement of what policies were decided there, since all emails so far have only discussed part of the issue, or been vague. It's hard to gauge whether python-support or python-central is more appropriate for the new policy without a clear statement of what that policy is. So here's *my* attempt at summarizing the BoF, based on your first mail and responses, and independent of the tools used: 1) Public modules and extensions should support all available Python versions, using a single binary package. 2) A new control field, XC-Python-Version will be used to determine what versions of Python a module supports. 3) The tight upper bound on module dependencies will be removed, provided the module actually works on future versions of Python. The upper bound on extension dependencies will not, because then they wouldn't work. 4) python2.x-* virtual packages are to be used only when necessary, but packages can provide them regardless. 5) Private modules and applications should use some way to support more than one Python version, if possible. Is this accurate? 1), 3), and 4) contridict your original email, but match what you told me this time. 4) I am still unsure of, because Steve and Matthias gave different answers, and your answer is confusing -- if they're desirable, let's make them required; if not, let's not pollute the (virtual) package namespace unless we need them for a specific reason. 2) is suspect. It's not required for multiversion support *in policy*. The working python-support implementation in unstable, while still lacking such fields, proves that. I feel it's much more of an implementation issue than a policy one, and so if we don't use python-central (which I gather is the impetus for it), we shouldn't bother with the control field either. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: python-central vs python-support
On Mon, 2006-06-05 at 14:23 +0200, Marc Dequènes wrote: > Steve Langasek <[EMAIL PROTECTED]> writes: > > > No. An extension has to be separately *compiled* for each python version > > it's to support. A python-foo package containing > > /usr/lib/python2.3/site-packages/foo/foo.so and > > /usr/lib/python2.4/site-packages/foo/foo.so must not claim to be compatible > > with python (>= 2.5). > > I'm not that silly... > > > However, it *should* be possible to provide a toolchain such that this > > python-foo can be binNMUed when python-2.5 becomes available and > > automatically pick up support for it. > > That's what i suggested in other words. If we can rely on python-all (or > any other mean to get the list of supported versions), and as we already > have the list of package-specific supported versions (via .version in > python-support for example), so that every version is compiled without > having to care to provide the list of versions manually, so > transitioning becomes seamless. CDBS could be enhanced to generate all > the necessary rules, so managing python packages would be painless. CDBS is not necessary; look at python-gst0.10's packaging. The versions it's built for is controlled entirely via a single Make variable, and it uses regular debhelper. This could be further refined to find all installed versions of Python at build time, and simplified because everything would be in the same binary package (so it wouldn't be limited by debian/control). New tools aren't needed to make binNMUs easier, though they can do that too. A few policy changes (mostly what I outlined in my last mail) would be enough to do that with existing tools and minimal trouble. New tools are mostly needed to prevent rebuilding at all (by fixing byte-compilation issues). -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: FTWCA python-central vs pyhton-support
On Thu, 2006-06-08 at 09:33 +0200, Raphael Hertzog wrote: > [ Please don't keep the BTS cced if you reply unless you want to express a > concern about the dh_python implementation suggested by the bug ] > > Hello everybody, > > Matthias has been working on integrating python-central support in > dh_python and submitted #370833: > > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=370833 > > This dh_python will use python-central for any binary package with a > XB-Python-Version: field. There's no optional python-support either. > > So anyone unhappy with python-central should really come up with another > dh_python implementation RSN. Previously we rolled back Joss's patch because Matthias didn't like it, and it took him several months to write a new one. Plus, Joss's patch actually worked, and formed the basis of a system that's now in common use (python-support). Since then, a new policy -- from Matthias -- has made that patch invalid. It's unfair (at best) to demand someone write a replacement Real Soon Now, when this *is* a replacement, and not soon at all, and pycentral got a head start because it designed the new policy. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: FTWCA python-central vs pyhton-support
Please don't Cc me. I am on the list. On Thu, 2006-06-08 at 23:05 +0200, Matthias Klose wrote: > Joe Wreschnig writes: > > On Thu, 2006-06-08 at 09:33 +0200, Raphael Hertzog wrote: > > > [ Please don't keep the BTS cced if you reply unless you want to express a > > > concern about the dh_python implementation suggested by the bug ] > > > > > > Hello everybody, > > > > > > Matthias has been working on integrating python-central support in > > > dh_python and submitted #370833: > > > > > > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=370833 > > > > > > This dh_python will use python-central for any binary package with a > > > XB-Python-Version: field. There's no optional python-support either. > > > > > > So anyone unhappy with python-central should really come up with another > > > dh_python implementation RSN. > > > > Previously we rolled back Joss's patch because Matthias didn't like it, > > no, not because I didn't like it, because it didn't work. It did work, it just didn't work like you wanted it to. python-support works. There are packages in the archive using it to support multiple Python versions and remove upper dependency bounds. > > Since then, a new policy -- from Matthias -- has made that patch > > invalid. It's unfair (at best) to demand someone write a replacement > > Real Soon Now, when this *is* a replacement, and not soon at all, and > > pycentral got a head start because it designed the new policy. > > If you start speaking about fairness, please consider that > python-support was uploaded to unstable without any discussion, with > dh_python forcing python-support for every rebuild of a package. I did consider that. We rolled back Joss's patch. Why should yours get special treatment? > In > #370833 Joss claims that "it was decided at debconf to use > python-support for python modules". That's plain wrong. I do not > understand why somebody tries to make a point by telling nonsense. I've heard some degree of conflicting claims from everyone who was at the BoF. Many packages are *already* using python-support for Python modules. So in that sense, it was decided long before the BoF to use python-support. > I didn't see any updates for python-support in the past months, Then you're blind or stupid. python-support 0.2 was uploaded on April 26th, 2006, and contained many changes requested by the RMs and maintainers. Joss made two maintenance uploads during May. It is being used by many Python packages already in unstable. It's not complete, and it's certainly not up-to-date with the specifications *you* want. But Joss is actively developing it, and to claim otherwise is incredibly disingenuous. > Joss refused to change python-support, Joss refused to change python-support to work like you wanted it to. > therefore I did move on with > python-central, which is now in unstable as well. "He started it!" Sorry, that argument doesn't work. Just because Joss rushed ahead with python-support in a way you didn't like doesn't mean you get to rush ahead with python-central (and python itself) in a way that several participants of this list have stated they don't like. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: Bug#370833: New dh_python proposal
On Sat, 2006-06-10 at 09:08 +0200, Raphael Hertzog wrote: > On Fri, 09 Jun 2006, Joey Hess wrote: > > Raphael Hertzog wrote: > > > Because Matthias and Josselin do not (or can't or don't want to) work > > > together. I tried my best to fill the gap. :-/ > > > > I appreciate your work, but I am not comfortable supporting two > > competing implementations in debhelper. > > Please reconsider. If you don't integrate that version now, we'll most > probably do the switch to python2.4 in the next days and we'll replace > dh_python calls by dh_python2 calls (integrated for example in the python > package) or we'll remove call to dh_python in favor of dh_pycentral (existing > and in python-central) / dh_pythonsupport (not existing but quick to > create). Please don't prefix non-debhelper commands with dh_ if you do this. Call it deb_pysupport/deb_pycentral or something, and likewise please don't mess with #DEBHELPER#. Stomping all over the namespace because you were rejected from the package per se seems to miss Joey's point. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: python-central vs python-support
On Fri, 2006-06-09 at 08:32 +0200, Raphael Hertzog wrote: > On Mon, 05 Jun 2006, Raphael Hertzog wrote: > > On Sun, 04 Jun 2006, Joe Wreschnig wrote: > > > policy is. So here's *my* attempt at summarizing the BoF, based on your > > > first mail and responses, and independent of the tools used: > > > > > > 1) Public modules and extensions should support all available Python > > > versions, using a single binary package. > > > > > > 2) A new control field, XC-Python-Version will be used to determine what > > > versions of Python a module supports. > > > > > > 3) The tight upper bound on module dependencies will be removed, > > > provided the module actually works on future versions of Python. The > > > upper bound on extension dependencies will not, because then they > > > wouldn't work. > > > > > > 4) python2.x-* virtual packages are to be used only when necessary, but > > > packages can provide them regardless. > > > > > > 5) Private modules and applications should use some way to support more > > > than one Python version, if possible. > > > > > > Is this accurate? 1), 3), and 4) contridict your original email, but > > > match what you told me this time. > > > > Yes, this is a good summary IMO, however I don't remember if we discussed > > point 5. > > Joe, could you integrate all this in the real document of the python policy? > Any other volunteer for that task? I will try to write something up Monday; I need to set aside some time to watch the BoF properly, and want to reread the pycentral/support code before I try to formalize the policy. If someone else wants to preempt me feel free. It also depends a little on how the implementation details play out, I guess. I can write the policy strictly independently of the tools, but the current policy has a lot of example code, and I'd like to keep that too. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: Status of recent and upcoming changes
On Sun, 2006-06-11 at 15:02 -0500, Graham Wilson wrote: > On Sun, Jun 11, 2006 at 05:50:06PM +0200, Matthias Klose wrote: > > - Some infrastructure was implemented to remove all hardcoded > >information about versions in the packaging scripts, so that > >sourceless package rebuilds (binary NMUs) are enough to update > >a package for new/removed/default python versions. > > Is it possible to elaborate on this point more in the final mail, or at > least provide a link so that maintainers can know how to use this new > funtionality? > > Joe, will this be documented in the new Python policy? As I said, I do want to keep the code examples. But until something is actually merged into debhelper I don't think it will help to document it in policy. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
New Python policy
I have updated the Python policy document based on the discussions on the list. An updated DebianDoc (and HTML and text files) can be found at http://people.debian.org/~piman/python-policy/. It: 0) Removes a lot of cruft/debate from the document itself 1) Explains X*-Python-Version 2) Explains that we don't want tight dependencies 3) Deprecates /usr/lib/site-python 4) Outlines using python-support/central with debhelper 5) Tries to cover more on private modules It does not: 1) Cover CDBS, since I don't know how to use it properly 2) Give examples for Python packages without using debhelper; I suspect those are now almost impossible to make sanely. Probably: 1) The "Upgrade Procedure" section needs work (from RMs) 2) I have some details wrong 3) My explanations could be clearer (but I feel it's an improvement over the old structure) -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: Deprecating /usr/lib/site-python in python policy
On Mon, 2006-06-12 at 17:08 +0300, Juha-Matti Tapio wrote: > On Mon, Jun 12, 2006 at 09:20:13AM +0200, Raphael Hertzog wrote: > > /usr/lib/site-python is on the sys.path of all python versions so we must > > make sure to provide the best (ie .py with working .pyc) for all python > > versions. > > Why "must"? > > Has this for some reason become a problem just recently or has this always > been a problem? I do not think missing or invalid .pyc's there could cause > noticable performance problems (especially since it is at the end of the > search path or at least near it) and certainly the modules are importable > from multiple python versions already. > > As a user I am greatful for all the people making an effort to improve > Debian Python, but I am beginning to worry if you people are taking too many > steps at once this close to the freeze. Only a dozen or so packages bother with /usr/lib/site-python, and almost all of them are buggy for placing unstable APIs in a public module directory anyway. Note that it's *not* being removed from sys.path (yet), it's just being deprecated. We deprecated /usr/doc with far more than a dozen packages still using it. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: Debhelper support should be ready tomorrow
On Mon, 2006-06-12 at 12:25 -0700, Steve Langasek wrote: > On Mon, Jun 12, 2006 at 09:06:43PM +0200, Raphael Hertzog wrote: > > On Mon, 12 Jun 2006, Raphael Hertzog wrote: > > > On Mon, 12 Jun 2006, Raphael Hertzog wrote: > > > > I would love to have this fixed by today's dinstall. That's why I > > > > prepared the attached NMU, please just send an OK and I'll upload it (of > > > > course, feel free to do it yourself in a maintainer upload). > > > > > Without any answer from you, I might upload it right before dinstall > > > > (depending on the opinion of other involved people). > > > > Just to avoid confusion, this would mean uploading between 18:00 and > > > 19:00 UTC > > > today (in about 9.5 hours). > > > Debhelper uploaded. We also had a python-central upload (fixing > > dh_pycentral) and a python-support (adding dh_pysupport). So hopefully we > > should be able to really start updating modules to the new policy tomorrow. > > > Build-Depends will have to require: > > - python (>= 2.3.5-7) (for pyversions script) Okay, I've updated the policy document to use that version in the Build Dependencies section. > Are packages expected to call pyversions directly, or is this something that > should be a versioned dep from one of the other build-dependencies? Some packages will call pyversions themselves to get the available list of Python versions, and run each one on setup.py (see [EMAIL PROTECTED] for an example). Pure Python modules using python-support won't need to. > Hmm, I guess it's called by dh_python, and debhelper doesn't depend on > python at all. Yes, this is already the cause of many FTBFS bugs in Python packages when developers don't test in pbuilder. I've added a note to the policy to this effect as well. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: New Python policy
On Tue, 2006-06-13 at 01:14 +0200, Matthias Klose wrote: > Joe Wreschnig writes: > > I have updated the Python policy document based on the discussions on > > the list. An updated DebianDoc (and HTML and text files) can be found at > > http://people.debian.org/~piman/python-policy/. > > > > It: > > 0) Removes a lot of cruft/debate from the document itself > > 1) Explains X*-Python-Version > > 2) Explains that we don't want tight dependencies > > 3) Deprecates /usr/lib/site-python > > 4) Outlines using python-support/central with debhelper > > 5) Tries to cover more on private modules > > > > It does not: > > 1) Cover CDBS, since I don't know how to use it properly > > 2) Give examples for Python packages without using debhelper; > > I suspect those are now almost impossible to make sanely. > > Some additions: > > - mention /usr/share/python/debian_defaults > - add paragraph about python packages > - describe the XB-Python-Version field > - add section about provides > - extend paragraph about byte compilation > - remove the paragraph about distutils > - enhance the section about support tools Merged and updated the site. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: Coordinated effort to update python packages
On Tue, 2006-06-13 at 22:05 +0200, Mike Hommey wrote: > On Tue, Jun 13, 2006 at 09:55:08PM +0200, Raphael Hertzog <[EMAIL PROTECTED]> > wrote: > > When you install some python extensions (.so), it is common that they come > > with > > associated .py files (modules). Those .py files usually are the same in > > /usr/lib/python2.3 and /usr/lib/python2.4, that's why python-central will > > keep only one copy of them in /usr/share/pycentral and symlink them back > > into > > /usr/lib/python2.[34]/ ... > > > > However if the .py installed in /usr/lib/python2.3 is different from the > > one installed in /usr/lib/python2.4, then you can't share it between both > > versions (and you need the DH_PYCENTRAL="nomove" variable). That what's I > > meant. Feel free to fix the wording in the wiki if you have a better way > > to express it. > > Okay, it sounds clearer said like that. Stupid question: why doesn't > pycentral autodetect if files are different or not ? I don't actually > see a reason why DH_PYCENTRAL need to be set... > > > > PS: Anyways, there's one package you didn't list because it's in NEW, > > > which WON'T be merged as required by the new policy: python2.x-xpcom, > > > provided by xulrunner. > > > The reason is simple: different installations for different python > > > versions CAN'T coexist without breaking each other. > > > > OK, then indicate "current" in XS-Python-Version and support only the > > current version in python-xpcom (make sure to generate the provides > > field). > > What about people who want to use pyxpcom with modules that aren't > available with the current python ? But a user couldn't install more than one such module at a time, since the packages are mutually exclusive. It seems far better to me to just force everyone to use one version and avoid the issue. Or, fix pyxpcom, which sounds horribly broken. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: Bugs have been filed, work can begin
On Wed, 2006-06-14 at 11:55 +0200, Raphael Hertzog wrote: > On Tue, 13 Jun 2006, Raphael Hertzog wrote: > > Here's a list of sources packages that need to be updated (266 packages): > > http://people.debian.org/~hertzog/python/sources-by-maint > > Pierre Habouzit (Madcoder) did the mass-bug filing: > http://bugs.debian.org/submitter:[EMAIL PROTECTED] > http://bugs.debian.org/usertag:debian-python@lists.debian.org > > And a wiki page to coordinate the NMUing: > http://wiki.debian.org/DebianPython/BSP > > Everybody's help is welcome now! The mass bug filing caught feedparser, gst0.10-python, gst-python, pygame, and mutagen. All of which I uploaded last night and will be entering unstable shortly. It missed pydance, uligo, and quodlibet, which I have not uploaded (quodlibet can't be until the transition, pydance and uligo I just haven't gotten around to yet). So far, it's not been very helpful. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: Python-Standards-Version
On Thu, 2006-06-15 at 13:44 +0200, Pierre Habouzit wrote: > Le jeu 15 juin 2006 13:36, Piotr Ozarowski a écrit : > > Pierre Habouzit ([EMAIL PROTECTED]): > > > > well the thing is there is no way to track all the packages that > > > *have* to follow the python subpolicy, and that makes the work of > > > tracking them for transition harder. > > > > > > I think/thought it makes sense that a source package lists all the > > > subpolicies it's supposed to follow, because it makes our lifes > > > easier. > > > > You can't bump Standards-Version number if your package is not > > following *all* standards (i.e subpolicies). > > > > The fact is that python policy is not official yet (at least document > > on d.o is not updated) and > > /usr/share/doc/debian-policy/upgrading-checklist.txt.gz > > doesn't mention about new python policy at all. > > we are not talking about the same target. My goal is to be able to spot > every package that has to conform to the python subpolicy by just > listing those fields. Obviously the Standards-Version is bumped also, > but it's not distinctive of python and not python related packages. This isn't really helpful now. Many packages have been uploaded already, without this field. Anything conforming to the new policy has XS-Python-Version. That seems to be enough to detect that it's been migrated. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: Bug#373387: python transition
On Thu, 2006-06-15 at 09:15 +0200, Pierre Habouzit wrote: > Le jeu 15 juin 2006 00:00, Peter Samuelson a écrit : > > unblock 373387 by 373628 > > thanks > > > > (This block was a false positive. While we use cdbs a little, we > > don't actually use it for anything python-related.) > > > > First, it's not clear to me what advantages anyone would get from > > "fixing" the subversion packaging to comply with current python > > policy. I do not see what is broken about our current package. It > > doesn't hard-code python 2.3 anywhere, so it's binNMUable. (Well, > > other than the "Provides: python2.3-subversion", which can probably > > go away now. It's left over from renaming the package from that.) One advantage of the new policy is that it automatically records the version of Python it was compiled against as XB-Python-Version, so whether or not the package needs a binNMU is more obvious (a dependency on a specific Python version doesn't tell you whether it's an accident of compiling against that version, or if it really needs that version). > > It's even less clear what point there is in supporting multiple > > simultaneous python versions, except to avoid the need for a binNMU > > once every 3 years. python-subversion has very few users. Building > > the module code twice, and making the package hard to backport to > > stable, is enough of a pain to make it reasonable to ask what benefit > > these users will get in return. Supporting just one version, the current one, is fine. However, you do need to migrate the package to use pycentral, since dh_python alone will no longer manage the byte (re)compilation of the .py modules. > please follow-up on debian-python@ Following up both places, I don't see a good reason to move the discussion. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: Quick poll on what's best
On Mon, 2006-06-19 at 19:24 +0200, Raphael Hertzog wrote: > Hello, > > as you may have read in the previous mail, the latest dh_python uses > a "debian/pyversions" file to find out the Python versions that the > packages supports but uses as fallback the XS-Python-Version field in > order to not break compatibility with packages that have already been > updated. > > For that I've written code in dh_python that re-creates the content of > debian/pyversions from the XS-Python-Version field. However it's now quite > clear that this XS-Python-Version field is only essential for > python-central and as such keeping that compatbility code inside dh_python > may be breaking the rule that dh_python tries to be agnostic (and which is > the wish of Joey Hess also). So you're removing one of the control fields while adding yet another one in your previous mail? No thanks, I'm done with this stupidity. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Orphaning Python packages
Hi everyone, I am no longer going to be involved in Python module packaging. I've orphaned feedparser and the GStreamer Python bindings (the 0.8 and 0.10 versions). These are used by several packages, and so I hope someone adopts them. feedparser is a pretty easy package, the GStreamer ones aren't. I'm willing to answer questions about the packages (and help whoever adopts the later with GStreamer in general) but I won't sponsor uploads. I've also put python-mutagen up for adoption. I'm upstream for this module, and intend to continue developing it. It's a simple package using distutils. If you're interested in maintaining it, let me know. I've not yet orphaned my Python applications. I may in the future; if you are interested in pydance (which doesn't run right now anyway, due to a regression in Pygame/SDL) or quodlibet please let me know. The Python policy SGML document is available at http://people.debian.org/~piman/python-policy, I suggest someone with more patience adopts it and finds a new canonical home for it. Please give the first bullet point in the "Upgrade Procedure" checklist careful thought before you decide to take this on. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: Updated dh_python to satisfy everybody
On Tue, 2006-06-20 at 06:49 +0200, Andreas Barth wrote: > * Raphael Hertzog ([EMAIL PROTECTED]) [060620 01:35]: > > On Mon, 19 Jun 2006, Raphael Hertzog wrote: > > > - uses "XS-Python-Standards-Version: 0.4" as reference field to run > > > in new > > >policy mode. The presence of XS-Python-Version will also trigger > > > the new > > >policy mode (this is for short-term compatibility, it may be > > > removed in > > >the not too-distant future). > > > > Joe proposed on IRC to use "debian/pycompat" instead of a new field. It > > sounds very much debhelper-ish and I like it. > > It depends what the field means. > > For me, the Standards-Version was mainly a marker to say "this package > is compatible to Version x.y of the policy" - which allows not only > debhelper to work on it, but also to search for old packages etc. This > is incompatbile with debian/pycompat (at least, if you want to do it > efficient). debhelper does not use Standards-Version for this purpose. It doesn't use it at all, as far as I know. debhelper uses a DH_COMPAT environment variable or debian/compat file. Not complying with the latest policy is a bug, regardless of what Standards-Version is declared. Not using the latest debhelper features is not a bug, as long as the package follows policy. Two fields for different purposes. X-Python-Standards-Version tried to overload one with the other. That's a recipe for disaster. Besides, eventually Python policy will be merged into real policy (hah hah, right) and then the field will exist only to give tool implementation details. Using X-PSV also misses the criticism of X[BS]-Python-Version, which is that dpkg's database should not be used for packaging tools in such a fashion (I'm not saying I agree with it -- but that's what Joss has said about X-PSV, and the fact that the new Python policy required a patch to dpkg I think gives it some legitimacy). To get rid of one while adding the other is dumb. Pierre Habouzit, the developer who suggested X-PSV, has told me in private that he agrees with my criticism, and is surprised that Raphael went ahead with using it before any discussion on the matter (besides a brief criticism from me, on the list, about why his intended purpose would be pointless). -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: Updated dh_python to satisfy everybody
On Tue, 2006-06-20 at 07:27 +0200, Andreas Barth wrote: > * Joe Wreschnig ([EMAIL PROTECTED]) [060620 07:14]: > > On Tue, 2006-06-20 at 06:49 +0200, Andreas Barth wrote: > > > * Raphael Hertzog ([EMAIL PROTECTED]) [060620 01:35]: > > > > On Mon, 19 Jun 2006, Raphael Hertzog wrote: > > > > > - uses "XS-Python-Standards-Version: 0.4" as reference field to > > > > > run in new > > > > >policy mode. The presence of XS-Python-Version will also > > > > > trigger the new > > > > >policy mode (this is for short-term compatibility, it may be > > > > > removed in > > > > >the not too-distant future). > > > > > > > > Joe proposed on IRC to use "debian/pycompat" instead of a new field. It > > > > sounds very much debhelper-ish and I like it. > > > > > > It depends what the field means. > > > > > > For me, the Standards-Version was mainly a marker to say "this package > > > is compatible to Version x.y of the policy" - which allows not only > > > debhelper to work on it, but also to search for old packages etc. This > > > is incompatbile with debian/pycompat (at least, if you want to do it > > > efficient). > > > > debhelper does not use Standards-Version for this purpose. It doesn't > > use it at all, as far as I know. debhelper uses a DH_COMPAT environment > > variable or debian/compat file. > > because there haven't been any such drastic changes in the normal policy > as we had in the python policy - and, compat means something *very* > different. What does it mean? This thread more or less started when Raphael updated dh_python to change behavior based on the presence/absence of the Python-Standards-Version field -- exactly analogous to what debhelper's compat levels are. If it was just a tool for the RMs to track the migration, I'd not have a problem with it[0], but the current proposal is that it actually affects the package build. That's bad, and at odds with the way the rest of debhelper (and all existing packaging tools) work. [0] Except for the fact it's useless, because the presence/absence of a XS-Python-Version tells the same thing. Which I said before. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: Orphaning Python packages
On Mon, 2006-06-19 at 20:20 +0200, Carlos Galisteo wrote: > Hello Joe. > > I could be interested about adopting feedparser since I use it in a couple > programs. > > Bad news are that I'm a really new mantainer and could need some initial > help (no problem about looking for a sponsor). > > If you have no problems about that I'll adopt it and start working on it > tonight. The package is orphaned, so if you want it, ITA it. I don't really have to walk someone through the basics of Python packaging though, sorry (especially since it's changing every other hour). The package is not at all complicated. I don't expect it to ever need changes if Python policy stablizes, since it's a single .py file with a distutils script. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
pycentral refuses to overwrite local files
When installing a package that uses pycentral, that I had previously installed by hand, I got an error: pycentral: pycentral pkginstall: already exists: /usr/lib/python2.3/site-packages/mutagen/__init__.py pycentral pkginstall: already exists: /usr/lib/python2.3/site-packages/mutagen/__init__.py dpkg: error processing python-mutagen (--install): subprocess post-installation script returned error exit status 1 I think making this an error is a bad decision. File conflicts within the packaging system should already be dealt with via dpkg. Outside of the package system, dpkg always silently overwrites local non-configuration files, and it's not an error normally. This is particularly bad for Python, because the default install prefix for local software is /usr, and there's no distutils uninstall command. So you have to manually find and remove all the files from any locally-installed package to switch to dpkg's version. Unless there's a good reason for this behavior, I think this is a grave bug for pycentral, since no other part of the packaging system creates errors in this situation. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: installation of scripts in the brave new Python world
On Tue, 2006-06-27 at 20:22 -0300, Gustavo Franco wrote: > On 6/27/06, Carlo Segre <[EMAIL PROTECTED]> wrote: > > > > Hello All: > > > > As I was trying to implement the new Python policy, I noticed an effect > > which is not directly addressed in the policy. That is when a package > > contains modules and scripts, there have to be special precautions taken > > to make sure that the scripts are not installed with a first line of > > > > #!/usr/bin/pythonX.Y > > > > instead of #!/usr/bin/python > > > > (...) > > How it's that different than patch the source packages to > s/python2.4/python/ as we do actually in some cases or replacing to > python2.3 when it isn't 2.4 compatible code? distutils rewrites scripts at installation time. Patching the source package will not work. http://docs.python.org/dist/node10.html -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: installation of scripts in the brave new Python world
On Wed, 2006-06-28 at 20:09 +0200, Josselin Mouette wrote: > Le mardi 27 juin 2006 à 19:10 -0500, Joe Wreschnig a écrit : > > > How it's that different than patch the source packages to > > > s/python2.4/python/ as we do actually in some cases or replacing to > > > python2.3 when it isn't 2.4 compatible code? > > > > distutils rewrites scripts at installation time. Patching the source > > package will not work. > > > > http://docs.python.org/dist/node10.html > > How about patching distutils in Debian to get rid of this bug? While I'm all for patching stupid upstream behavior, it would be a pretty big API deviation. There's a relatively simple fix, which is using /usr/bin/python for the last run (or even just for an install_scripts run). -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: dpkg-repack warnings: what effect?
On Tue, 2006-07-25 at 02:07 +0200, Adeodato Simó wrote: > * Matthias Klose [Tue, 25 Jul 2006 00:56:29 +0200]: > > > > > I agree that having Python-Version in binary packages is dislikeable. > > > > Out of curiosity, Matthias, would be a DEBIAN/pyversions file able to > > > > provide the same information to pycentral? >^^^ > > > just looking at the Packages (and the Sources) files. > > Yeah, I knew this was the reason for putting the info in the headers, I > understand that. But I was only asking if pycentral itself would have > enough with DEBIAN/pyversions, which is different from implying that I'd > rather have pycentral use D/pv at the cost of losing the benefits of > having that info in the Packages file. ;-) > > I assume the answer to this question is yes, yes? > > Now, I have a sequence of events that have left me puzzled: > > - After writing the above, I intended to ask why XB-Python-Version was > necessary in the Packages file, if that information was already in > Sources via XS-Python-Version. > > - But then I thought: "d'oh, obvious: XB-P-V must be an _expanded_ > version of XS-P-V, with a plain list of python version the package > was built for, and this is much easier to parse than an expresion > with >= and <<, plus only with XS-P-V you can't know e.g. if a > binNMU after dropping 2.3 was successful or not". > > - But after that, I quickly discover that my assumption is wrong, and > the archive contains a number of binary packages whose Python-Version > header contains [<=>], and it's in fact the same as P-V in the > source package. If the package contains binary modules, it should only contain the explicit list of versions it was compiled for. But if the package does not contain any binary modules, XS-P-V and XB-P-V will be identical, since nothing gets "built" and new versions can be supported without a binNMU. There is no reason this information can't, in theory, be contained in an installed file like DEBIAN/pyversions. You'd need to use the lintian lab or something similar to track rebuilds, but you'd not seriously lose any features. But please, don't make any more "cosmetic" feature/change requests until after the transition actually happens. (Or, finally announce that Python 2.4 is plain too late for etch. Then people can have as long as they want to debate over this crap, and Python developers can just give up hope.) -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: dpkg-repack warnings: what effect?
On Tue, 2006-07-25 at 14:27 +0200, Josselin Mouette wrote: > Le lundi 24 juillet 2006 à 23:18 -0500, Joe Wreschnig a écrit : > > (Or, finally announce that Python 2.4 is plain too late for etch. Then > > people can have as long as they want to debate over this crap, and > > Python developers can just give up hope.) > > Well, isn't the whole purpose of that new policy stuff to delay python > 2.4? Oh, so it's not possible for someone to be an irresponsible maintainer without being an Ubuntu shill as well? When did they hire you, then? (Yes, this message isn't constructive, but I'm beyond caring.) -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: Generation of "python" dependencies for public extensions versus python2.3
On Tue, 2006-07-25 at 22:07 +0200, Loïc Minier wrote: > Hi, > > While trying to install the packages from experimental where python > is python2.4, I was surprized by the non-upgradability of the > python-gmenus package (from gnome-menus): it is built with > XS-Python-Version set to current, and ends up with: > Depends: libc6 (>= 2.3.6-6), libglib2.0-0 (>= 2.10.0), > libgnome-menu2 (>= 2.14.0), python-central (>= 0.5), python (>= > 2.3), python (<< 2.4) > Python-Version: 2.3 > > Since "current" was 2.3 when it was built, it ships files below > /usr/lib/python2.3, so why doesn't it end with a python2.3 dependency > instead of pure python dependencies? And why does it "conflict" with a > "python" 2.4? The scripts in /usr/lib/python2.3/site-packages/GMenuSimpleEditor call plain "python". That's probably causing the dependency. There's another bug in the gnome-menus package too: $ dpkg -s gnome-menus | grep Depends Depends: libc6 (>= 2.3.6-6), libglib2.0-0 (>= 2.10.0), libgnome-menu2, python (>= 2.3), python (<< 3), python-gmenu (= 2.14.0-3), python-glade2 So gnome-menus claims to work with 2.4. But in the package /usr/bin/gmenu-simple-editor has: #!/usr/bin/env python ... if not "/usr/lib/python2.3/site-packages" in sys.path: sys.path.insert (0, "/usr/lib/python2.3/site-packages") In other words, it will load the 2.3 extension even when it's running on Python 2.4. That's bad. Why hasn't this been updated to support multiple versions of Python? That's what public extensions should do now. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
Re: The new python policy and packaging
ing from section 2.5. There > should be something about providing pythonX.Y-foo if your package > provides an extension for version X.Y of python. Yes, this is a real mistake, probably my fault. The second paragraph can probably just be deleted. The Provides: has nothing to do with extensions. It's solely to do with whether or not version-specific modules or programs need to use your module. It can be appropriate for pure Python modules, or pointless for extensions that are not required by anything version-specific. > ps: I think policy should state clearly how it expects the various > fields in the control file to be initialized, rather than leaving it > mostly unstated and deferring to an external helper program, but that > is another thread Until dh_python's behavior stops changing every other week this is an act of futility. I tried to emphasize this previously, but it seems to be a lost cause. Packaging Python programs is going to require debhelper or CDBS, as well as pysupport or pycentral, for the forseeable future. I realize this will make you very unhappy, but it's just something you will have to deal with for now. -- Joe Wreschnig <[EMAIL PROTECTED]> signature.asc Description: This is a digitally signed message part
