Re: new dh_python proposal
Hello, Piotr. In short - your post is long. Brain is limited. Better place an overview in wiki and discuss in parts. Before discussing your proposal I would really appreciate if somebody from insiders could describe situation in Debian Python in all possible detail including the history of development. I believe that this piece of work is absolutely necessary, because the problem of repackaging Python packages for Debian and maintaining integrity between installed packets, packages and Python versions is too complex to keep every detail in mind unless you can devote your full time to the problem. One way to make the status more obvious is to answer to emails in this group, and merge answers into wiki, so that the next time you can point a person to the exact place of documentation. You say that attempt to merge -cental and -support failed, but didn't mention why. You say that current tools have problems that occur at install/upgrade time, but do not mention these problems explicitly. So it is impossible to say whatever your ideas solve current problems and won't add new. Referencing rtinstall/rtupdate/rtremove scripts is cool for a seasoned Debian developer, but for a Python developer it means nothing. In other words - not only Python things are obscure for Debian developers, but Debian stuff is also a mystery for Python masters, so these things should be explained symmetrically for both communities. The chances for Debian Python Packaging experts to pop up are magnitude less if we won't explain the situation in detail. Now about the proposal (from newcomer's point of view): dh_python is a shell script -- I have a strong belief that Python package automation scripts should be written in Python, there is no need to learn bashes when you program Python - do not expect that package maintainers will be able to debug their scripts in shell language. The description that dh_python will do at a build time looks like a solution, but again - it doesn't specify what problem is being solved. Perhaps I expected to see more high-level definition. May I ask a question - is there a difference between installation of Python Modules and Python Applications? Does the problem set you are trying to resolve with this proposal include the difference? Have you considered distributing Python Applications via virtualenv? How this proposal handles virtualenv installations? In conclusion my opinion is that problem set is not defined well enough to propose a solution or estimate if it will be effective both for current flow and for future ideas. I would start from wiki. --anatoly t. On Sun, Aug 2, 2009 at 6:53 AM, Piotr Ożarowski wrote: > My attempt to merge python-central and python-support failed few months ago, > so > here's another proposal to improve the Python situation in Debian. It's > nothing new > actually, just a compilation of previous Matthias' and Joss' ideas and few of > my > own. > > The main idea is to ship symlinks (not .pyc files!) in binary packages as > Matthias proposed back in February and thus get rid of both helper tools (most > problems with current tools occur at install / upgrade time). I want to keep > it > as simple as possible and let maintainers customize it in maintainer or > rtinstall/rtupdate/rtremove scripts (if really needed). > > Advantages: > * a lot less opportunities to break a system while installing / upgrading > Python packages, > * no more problems with packages that provide the same namespace and use > different helper tool, > * Python modules available out of the box (think about daemons), > * no more "which tool should I use?" or "how do they differ?" questions ;-) > > The main disadvantage of this approach is that architecture independent > packages > will have to be rebuilt once new Python version will be added to the > supported ones > (I think we can avoid a rebuild if a version is removed from the supported > ones). > Luk (our release manager) told me that binNMUs for arch:all packages will be > possible in Debian at some point, but it's a matter or months or even years, > so > for now - manual NMUs will be needed. I think it will be easy to detect which > packages need one basing on Contents file and Depends field. > > Another disadvantage is the fact that pysupport's namespace feature will not > be > supported. If we want to keep it simple and let dpkg know about all the files > - > new *-common package with all common __init__.py (and other if needed) files > will have to be provided, just like we did before pysupport 0.7. Otherwise > removing .pyc files will not be a trivial thing to do. Note that there are > lots > of problems with this feature anyway (upstreams tend to use site-packages for > things that don't belong there), so removing it will improve the situation at > the end, IMHO. > > I'll try to make dh_python a drop in replacement for dh_pycentral and > dh_pysupport (i.e. tools used at build time) for most packages, but I don't > consider
Re: new dh_python proposal
On Tue, Aug 04, 2009 at 12:00:09PM -0400, anatoly techtonik wrote: > Now about the proposal (from newcomer's point of view): > dh_python is a shell script -- I have a strong belief that Python > package automation scripts should be written in Python, there is no > need to learn bashes when you program Python - do not expect that > package maintainers will be able to debug their scripts in shell > language. [skipping the rest of your mail] I got the impression that dh_python will be written in Piotr's favourite language, Python, but may need to be rewritten in Perl for inclusion in debhelper: http://lists.debian.org/debian-python/2009/08/msg6.html Kumar -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Re: new dh_python proposal
[anatoly techtonik, 2009-08-04] > Before discussing your proposal I would really appreciate if somebody > from insiders could describe situation in Debian Python in all > possible detail including the history of development. I believe that > this piece of work is absolutely necessary, because the problem of > repackaging Python packages for Debian and maintaining integrity > between installed packets, packages and Python versions is too complex > to keep every detail in mind unless you can devote your full time to > the problem. Short? Our goal is to support more than one Python version at the same time if having one Python in the archive is not possible (f.e. when there are applications that still require older Python). We do that currently by shipping Python extensions for all supported Python versions in the same package and byte compiling Python modules at install time for all installed (and supported) Python versions (sharing .py files whenever possible). Read [1] and then [2] if you want more details (yeah, [1] is not official[3] as there were some differences of opinion :-) - I hope it will be easier to finally update the official policy if this proposal will be accepted) [1] http://people.debian.org/~srivasta/manoj-policy/ [2] http://lists.debian.org/debian-python/2009/02/msg00061.html [3] http://www.debian.org/doc/packaging-manuals/python-policy/ > You say that attempt to merge -cental and -support failed, but didn't > mention why. attempt is here[4,5], the reason why it failed is here[6] [4] http://lists.debian.org/debian-python/2009/02/msg00083.html [5] http://lists.debian.org/debian-python/2009/02/msg00099.html [6] http://lists.debian.org/debian-python/2009/03/msg00015.html > You say that current tools have problems that occur at > install/upgrade time, but do not mention these problems explicitly. So > it is impossible to say whatever your ideas solve current problems and > won't add new. see [7] and [8] for some of them. I tried to help many users (including co-workers and #debian-python channel guests) that had ImportError problems I couldn't reproduce on my machine and usually the solution was to reinstall related Python packages (due to missing symlinks or symlinks from older package versions not properly removed at upgrade). I had that problem once on my own system as well, couldn't reproduce it later (and thus couldn't really write a sensible bug report). Right now upgrading from Lenny's pycentral based packages will fail if list of files will be different in Squeeze' version (due to moving to pysupport or dropping "nomove" feature or simply by providing different module names) unless you provide a preinst script that will remove old files. Another issue is the fact that packages with common namespace have to use the same tool (search f.e. for twisted issues in the mailing list or BTS) as otherwise they will end in different directories (i.e. will be reachable from different locations of sys.path) and Python will not recognize that. Yet another issue is the pysupport namespace feature that actually causes more problems than it solves (see f.e. #459446 or #536739) Also, during upgrades, daemons downtime should be as short as possible and since Python modules are right now not available after unpacking debs, it's hard to accomplish it. You also have to do some extra work to be be able to start them during installation. [7] http://bugs.debian.org/python-central [8] http://bugs.debian.org/python-support > Referencing rtinstall/rtupdate/rtremove scripts is cool for a seasoned > Debian developer, but for a Python developer it means nothing. In > other words - not only Python things are obscure for Debian > developers, but Debian stuff is also a mystery for Python masters, so > these things should be explained symmetrically for both communities. > The chances for Debian Python Packaging experts to pop up are > magnitude less if we won't explain the situation in detail. see chapter 6[9] of "Manoj's policy"[1] [9] http://people.debian.org/~srivasta/manoj-policy/x673.html > Now about the proposal (from newcomer's point of view): > dh_python is a shell script -- I have a strong belief that Python > package automation scripts should be written in Python, there is no > need to learn bashes when you program Python - do not expect that > package maintainers will be able to debug their scripts in shell > language. I will write them in Python, old dh_python is written in Perl /bin/sh as dh_python's shebang is not that bad idea, BTW :-) [...] > May I ask a question - is there a difference between installation of > Python Modules and Python Applications? yes, Python applications should be installed in private locations (i.e. outside site-packages) whenever possible (to avoid name conflicts and unnecessary symlink burden). > Does the problem set you are > trying to resolve with this proposal include the difference? Have you > considered distributing Python Applications via virtualenv?
Modules with changing APIs/ABIs
Hi,
python-sip4 (the runtime support library beneath all bindings of Qt for
Python) changes its ABI with more or less every major release, and
sometimes between minor releases. While practically no code uses it
directly, all Python extension modules using Qt or KDE depend on it and
have to be recompiled for every new release.
So far, in python-qt{3,4} I have handled this by depending on the
current major version (i.e. python-sip4 (>= 4.x), python-sip4 (<< 4.(x
+1)), but several more project have started using it and this approach
clearly doesn't scale.
So far, I could think of two solutions:
1. Changing binary package names
This is more or less how libraries are handled: The runtime module is
shipped in python-sip4-x (with x being whatever the ABI version happens
to be) and changed appropriately.
2. Provides:
In this scenario, the package name stays the same, but python-sip4
provides sth. like sip-runtime-x.y, which is updated accordingly.
Packages like python-qt4 then have to depend on the correct sip-runtime
version.
In any case, exact version should not be hardcoded, but provided by a
substvar. A package which builds Python extension modules has to
build-depend on sip4 (which is the code-generator package) and invoke
dh_sip4 in its rules file. Each binary package that ships a Python
extension that depends on the sip module must then depend on
${sip:Depends}, which contains either the binary package name or the
virtual package name. Thus, the packages are binNMU-safe and a new
version of sip can be handled by a simple rebuild (assuming that the
binding code does not have to be adapted, which happens frequently
enough).
I would somehow prefer solution no. 2 (less hassle, no new queue
involve), especially since the module name is always sip.so and thus
python-sip4-x+1 replaces python-sip4-x completely anyway.
If you have any comments/thoughts, I'd be pleased to hear your input.
best,
Torsten
--
.: Torsten Marek
.: http://shlomme.diotavelli.net
.: tors...@diotavelli.net -- GnuPG: 1024D/A244C858
signature.asc
Description: Dies ist ein digital signierter Nachrichtenteil
Re: Existing modules for new Python
On Thu, 30 Jul 2009 17:33:45 -0400, anatoly techtonik wrote: > Can anybody tell what happens with existing Python modules when a new > Python version is deployed on the system? > Are they recompiled or reinstalled or left inaccessible in this version? They are left on the system in /usr/lib/pythonX.Y/site-packages or wherever they got installed. The new version of python goes into /usr/lib/pythonX.Z so the old ones won't be seen. They are still accessible via the old python version, but not accessible to the new python version. David -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
