continuous integration/testing for python packages [Was: Is it worth back porting PEP 3147...]
Hi, [discussion started at http://lists.debian.org/debian-python/2010/04/msg00046 should we continue or trim some of the cc'ed lists ?] On Mon, Apr 26, 2010 at 06:41:16PM -0400, Barry Warsaw wrote: > On Apr 26, 2010, at 06:35 PM, Nicolas Chauvat wrote: > >On Thu, Apr 22, 2010 at 01:52:11PM -0400, Barry Warsaw wrote: > >> How much of the transition testing is automated? It would be very > >> interesting > >> for example, to have a test framework that could run any combination of > >> Python > >> packages against various versions of Python, and get a report on the > >> success > >> or failure of it. This may not be a project for the distros of course - I > >> think upstream Python would be very interested in something like this. For > >> example, a tool that grabbed packages from the Cheeseshop and tested them > >> against different versions would be cool. If snakebite.org ever gets off > >> the > >> ground, that might be the best place to put something like this together > >> (though we'd care less about OSes that aren't Debian and Ubuntu). > > > >Unfortunately, Logilab does not have much man-power to offer to set > >this up at the moment, but would something like > >http://apycot.hg-scm.org/ fit your description of a test framework ? > > That's for continuous integration of Mercurial, right? Yes. > >We also have it running at logilab.org and cubicweb.org of course: > >http://www.logilab.org/view?rql=testconfig&vid=summary > >http://www.cubicweb.org/view?rql=testconfig&vid=summary > > > >As you can see with these second and third links, tests include > >lintian and piuparts checks. > > > >Is it something like this that you had in mind? > > Yes. What are you using to drive this? I'm not really up on CI tools, but > Hudson has been getting a lot of buzz. > > http://hudson-ci.org/ We are using http://www.logilab.org/project/apycot that is GPL software mainly developed and maintained by Logilab, but slowly reaching out to a larger audience. It uses a web framework to store the information in a db and provide a web user interface, plus slave testing bots running on one or more hosts that get the next task from the queue, execute it and store the results in the db. > What I like about your display is that a failure in one area does not > necessary mean a failure elsewhere. That way you can better see the overall > health of the package. You may find interesting the following blog posts about apycot and ways to display its information http://bit.ly/9dZQQE > as nearly automatic and effortless packaging in Debian and Ubuntu. We tried fully automatic packaging of Python programs years (8?) ago and did not succeed for distutils and setuptools were too far away from Debian packaging concerns. Introducing in mypackage/__pkginfo__.py and mypackage/setup.py all the information needed to generate the debian/* files without the need to modify them eventually meant more or less copying their whole content, for their is actually not much to generate. It also meant using a less efficient toolchain because of the added conversion step. We moved to having tools that check the consistency of the information provided by __pkginfo__ and debian/* files and make it easier to build the Debian packages. These tools are http://www.logilab.org/project/logilab-devtools Packaging a piece of Python software now requires a bit of (easy) work at first, but following releases only need one or two commands. And all the dh_python* helper scripts reduced that work even further. > What I have in mind is defining a set of best practices, embodied as much as > possible in tools and libraries, that provide carrots to Python developers, so > that if they adhere to these best practices, the can get lots of benefits such > ... > It's things like 'python setup.py test' just working, and it has an > impact on PyPI, documentation, release management, etc. These best > practices can be opinionated and simple. If they cover only 80% of > Python packages, that's fine. Developers would never be forced to > adhere to them, but it would be to their advantage to do so. Sounds good to me :) -- Nicolas Chauvat logilab.fr - services en informatique scientifique et gestion de connaissances Full link to blog posts: http://www.cubicweb.org/view?rql=Any+X+WHERE+X+is+BlogEntry%2C+T+tags+X%2C+T+name+%22apycot%22 -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100427080128.gc4...@volans.logilab.fr
Python2.4 has been removed, now what for Zopistas and Plonistas?
Hi, unfortunately, I've missed decision to remove Python2.4 from Debian, but the result is that Zope2.10, required for Plone 3.x, will not run on Squeeze anymore, and that the zope2.10 package from sid is now completely uninstallable. Even installing Zope from upstream's source, imho "the" recommended method, does no longer work without prior manual installation of Python and a load of additional packages from their respective upstream's source, too (makes it much more of a hassle...). I am not sure that this was a good choice, although I feel the pain of maintaining a Python 2.4 stack in Squeeze, because: * Plone, and thus Zope is still a major Python application, and I heavily doubt that Plone 4.0 will see the light in time to make it into Squeeze (it's currently in alpha state, afaik). * The upgrade from Plone 3 to Plone 4 requires significant work, which many users will probably not even be able to handle shortly, so their options will be to continue to run Lenny, compile from source entirely, along with appropriate testing and deployment efforts, or to switch platforms. * I therefore think that Plone 3 is here to stay for at least one or two more years, because of the migration effort. * Plone 3 absolutely requires Zope 2.10. If Python2.4 will stay out of Squeeze, I think removal bugs should be filed against all packages that are required for Plone3 and not elsewhere, including Zope 2.10 and friends to avoid trying to eg. install a package only to see an error message because depenencies can't be met. Right? Wrong? Kind regards, --Toni++ -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100427091730.15584.qm...@oak.oeko.net
Re: Python2.4 has been removed, now what for Zopistas and Plonistas?
Il 27/04/2010 11.17, Toni Mueller ha scritto: > I am not sure that this was a good choice, although I feel the pain of > maintaining a Python 2.4 stack in Squeeze, because: > > * Plone, and thus Zope is still a major Python application, and I >heavily doubt that Plone 4.0 will see the light in time to make it >into Squeeze (it's currently in alpha state, afaik). > > * The upgrade from Plone 3 to Plone 4 requires significant work, which >many users will probably not even be able to handle shortly, so >their options will be to continue to run Lenny, compile from source >entirely, along with appropriate testing and deployment efforts, or >to switch platforms. > > * I therefore think that Plone 3 is here to stay for at least one or >two more years, because of the migration effort. > > * Plone 3 absolutely requires Zope 2.10. When dealing with python2.4 removal, we asked Fabio about state of the art of Zope 2.14 and Plone 4. He replied he hadn't way to look at them at that time, and he would have looked upgrading the whole stack in April/May. CCing him, to see if things evolved. > If Python2.4 will stay out of Squeeze, I think removal bugs should be > filed against all packages that are required for Plone3 and not > elsewhere, including Zope 2.10 and friends to avoid trying to eg. > install a package only to see an error message because depenencies > can't be met. Right? Wrong? When removing python2.4 packages, we told ftpteam to leave rdependencies alone for the time being, as they have to be fixed, and clearing NEW again wouldn't have been that efficient. If those packages are beyond any repair, maybe filing RM bugs against them is the way to go. Fabio, thoughts? Regards, -- .''`. : :' : Luca Falavigna `. `' `- signature.asc Description: OpenPGP digital signature
Re: Python2.4 has been removed, now what for Zopistas and Plonistas?
* 2010-04-27 15:16, Luca Falavigna wrote: > When dealing with python2.4 removal, we asked Fabio about state of the > art of Zope 2.14 and Plone 4. He replied he hadn't way to look at them at > that time, and he would have looked upgrading the whole stack in > April/May. CCing him, to see if things evolved. The current stable release of Plone requires python2.4 and Zope2.10, which we cannot support in squeeze. I see no way to really support them, so I'd prefer to have the packages removed. > If those packages are beyond any repair, maybe filing RM bugs against > them is the way to go. Fabio, thoughts? I agree, feel free to file RM bugs for all the zope-related packages in the archive which depend on python2.4. Best regards, Fabio -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100427134245.gh26...@mail.tranchitella.eu
Re: continuous integration/testing for python packages [Was: Is it worth back porting PEP 3147...]
On Apr 27, 2010, at 10:01 AM, Nicolas Chauvat wrote: >[discussion started at >http://lists.debian.org/debian-python/2010/04/msg00046 >should we continue or trim some of the cc'ed lists ?] Is there a better place to discuss Python packaging issues that are of interest to both Debian and Ubuntu? Are all Ubuntu developers who care about shared Python concerns generally members of debian-python? I don't know these things. :) >We are using http://www.logilab.org/project/apycot that is GPL >software mainly developed and maintained by Logilab, but slowly >reaching out to a larger audience. There's also buildbot of course, which I guess it the granddad of Python CI tools, kind of. >It uses a web framework to store the information in a db and provide a >web user interface, plus slave testing bots running on one or more >hosts that get the next task from the queue, execute it and store the >results in the db. Are there things like a API (REST or otherwise) for pulling data out of apycot? >> What I like about your display is that a failure in one area does not >> necessary mean a failure elsewhere. That way you can better see the overall >> health of the package. > >You may find interesting the following blog posts about apycot and >ways to display its information http://bit.ly/9dZQQE > >> as nearly automatic and effortless packaging in Debian and Ubuntu. > >We tried fully automatic packaging of Python programs years (8?) ago >and did not succeed for distutils and setuptools were too far away >from Debian packaging concerns. > >Introducing in mypackage/__pkginfo__.py and mypackage/setup.py all the >information needed to generate the debian/* files without the need to >modify them eventually meant more or less copying their whole content, >for their is actually not much to generate. It also meant using a less >efficient toolchain because of the added conversion step. I'm not familiar with __pkginfo__.py, but I do agree that setup.py makes automated packaging difficult. We need a declarative syntax that can be consumed by more tools, which is why I'm so excited about Tarek's work in distutils-sig. >We moved to having tools that check the consistency of the information >provided by __pkginfo__ and debian/* files and make it easier to build >the Debian packages. These tools are >http://www.logilab.org/project/logilab-devtools Is there a wiki or online documentation documenting these tools, or is it all in the source? >Packaging a piece of Python software now requires a bit of (easy) work >at first, but following releases only need one or two commands. And >all the dh_python* helper scripts reduced that work even further. Is that easy work manual or automated? What does it take to Debianize random-simple-pypi-package? (By that I mean "run a script" or "inspect setup.py and write the debian/*" or "...?". >> What I have in mind is defining a set of best practices, embodied as much as >> possible in tools and libraries, that provide carrots to Python developers, >> so >> that if they adhere to these best practices, the can get lots of benefits >> such >> ... >> It's things like 'python setup.py test' just working, and it has an >> impact on PyPI, documentation, release management, etc. These best >> practices can be opinionated and simple. If they cover only 80% of >> Python packages, that's fine. Developers would never be forced to >> adhere to them, but it would be to their advantage to do so. > >Sounds good to me :) Right now, it's just an idea. There are a few existing attempts at this, so it's worth looking at what exists first. -Barry signature.asc Description: PGP signature
Re: Python2.4 has been removed, now what for Zopistas and Plonistas?
Hi, On Tue, 27.04.2010 at 15:42:45 +0200, Fabio Tranchitella wrote: > The current stable release of Plone requires python2.4 and Zope2.10, which > we cannot support in squeeze. I see no way to really support them, so I'd > prefer to have the packages removed. from my point of view, only Python 2.4 and a few supporting packages like python-setuptools, python-virtualenv, python-imaging, python-celementtree and python-elementtree, maybe python-crypto and a few others are really useful for running Plone 3.x, but there is imho no need to support the actual Zope packages in the system. I've not yet made an effort to figure out a minimally useful set of packages, though. My preferred outcome would be to have these Python packages in Squeeze, but remove everything else that is Zope 2.10 related, as these things are imho better installed using buildout, anyway. > > If those packages are beyond any repair, maybe filing RM bugs against > > them is the way to go. Fabio, thoughts? > > I agree, feel free to file RM bugs for all the zope-related packages in the > archive which depend on python2.4. +1 Kind regards, --Toni++ -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100427162001.11566.qm...@oak.oeko.net
Re: Python2.4 has been removed, now what for Zopistas and Plonistas? [ADDENDUM]
Hi, I forgot to ask what'd be the best way to go forward for Plonistas? I really think that there should be some time for Plonistas to upgrade, which the current scheme does not allow for. I see the following options: * Compile everything from source, including Python 2.4 So far, Debian's Python packaging in Lenny made it very easy to run Plone, much easier than on most other platforms. I'd like to keep it that way. * Provide a set of Python 2.4 packages, maybe via deb-ports, or via someone's home page * Provide a set of Python 2.4 packages via 'main' I'm not sure about how much the changes in the infrastructure would be an obstacle, or prevent this from happening (didn't check yet), and I don'tknow how the group thinks about the political and/or personal issues involved, either. What I do see is that it will most likely almost impossible to provide security support for Python 2.4 if upstream decides to stop providing these (Barry?). If there are no principal objections, I might try to adopt one or other of these packages. Kind regards, --Toni++ -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100427163213.12134.qm...@oak.oeko.net
Re: Python2.4 has been removed, now what for Zopistas and Plonistas? [ADDENDUM]
On Apr 27, 2010, at 06:32 PM, Toni Mueller wrote: >I'm not sure about how much the changes in the infrastructure would be >an obstacle, or prevent this from happening (didn't check yet), and I >don'tknow how the group thinks about the political and/or personal >issues involved, either. What I do see is that it will most likely >almost impossible to provide security support for Python 2.4 if >upstream decides to stop providing these (Barry?). Officially, Python 2.4 is EOL. The last release was 2.4.6 on 2008-12-19 and it was a source-only release. Even security patches will only be provided if there's someone motivated enough to do it, which for a release as old as 2.4 is doubtful. -Barry signature.asc Description: PGP signature
Re: Python2.4 has been removed, now what for Zopistas and Plonistas? [ADDENDUM]
* 2010-04-27 18:33, Toni Mueller wrote: > I forgot to ask what'd be the best way to go forward for Plonistas? > I really think that there should be some time for Plonistas to upgrade, > which the current scheme does not allow for. I personally use the unified installer, and it works fine; I liked more the debian packages, but it is quite complex to support them so I switched to what is considered by upstream the "only" way to install plone nowadays. Best regards, Fabio -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100427181109.gi26...@mail.tranchitella.eu
Re: continuous integration/testing for python packages [Was: Is it worth back porting PEP 3147...]
On Tue, Apr 27, 2010 at 11:07:51AM -0400, Barry Warsaw wrote: > On Apr 27, 2010, at 10:01 AM, Nicolas Chauvat wrote: > There's also buildbot of course, which I guess it the granddad of Python CI > tools, kind of. Yes, buildbot is the old-timer and has loads of features. As far as I know, extracting information out of buildbot is more difficult than it is with a tool like Apycot that's built on top of CubicWeb. > Are there things like a API (REST or otherwise) for pulling data out of > apycot? Sure, that's a basic functionnality of CubicWeb. For example, just take any url and append vid=xml http://apycot.hg-scm.org/projectenvironment/hg/full/108464 becomes http://apycot.hg-scm.org/projectenvironment/hg/full/108464?vid=xml If the data that's extracted isn't what you expect, file a ticket at logilab.org/project/apycot and the 'xml' view will be enhanced. As you could read from http://www.cubicweb.org/blogentry/779839 the views you apply to data sets are not restricted to producing html or xml, they can also directly produce json or pdf, graphs, etc. Other examples of the view mechanism would be the ones that use a specific vocabulary as in http://www.logilab.org/project/apycot?vid=doap or http://www.cubicweb.org/blogentry/779839?vid=sioc or http://www.cubicweb.org/cwuser/nchauvat?vid=foaf In short, CubicWeb was *designed* to publish its data under reusable formats, html being just one way to present data. Maybe we can restrict this part of the discussion to the cubicweb mailing list ? > I'm not familiar with __pkginfo__.py, __pkginfo__ is a declarative format used at Logilab. http://hg.logilab.org/logilab/devtools/file/tip/doc/pkginfo_variables.txt > We need a declarative syntax that can be consumed by more tools, > which is why I'm so excited about Tarek's work in distutils-sig. I agree, Tarek's work is a great improvement over the current situation. > Is there a wiki or online documentation documenting these tools, or is it all > in the source? It is mainly in the source. You will find several people that will be happy to help if you ask your questions on python-projects at lists.logilab.org. > Is that easy work manual or automated? What does it take to Debianize > random-simple-pypi-package? (By that I mean "run a script" or "inspect > setup.py and write the debian/*" or "...?". Easy manual work. I'm cc'ing the people that do it often so that they can provide details. Alexandre, Sylvain ? -- Nicolas Chauvat logilab.fr - services en informatique scientifique et gestion de connaissances -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100427200947.ga15...@volans.logilab.fr
