Build-time testing
Hello. Many Python modules ship unit tests which do not require module instalation or user interaction and so can be run automatically during the build time. They can be a very good QA tool, but it looks like they are not used in all packages. python-pylons: not used python-pybabel: not used pylint: used, but produce errors which are ignored python-sphinx: apparently used In many cases $PYTHON setup.py test (for all supported $PYTHON's) is enough to run the tests and any errors mean that something is wrong with the environment (provided the tests are correct). Shouldn't we have some written recommendations (or even policy) on this matter? -- WBR, wRAR signature.asc Description: Digital signature
Re: Build-time testing
FWIW I usually prefer to run unittests against (PYTHONPATH) installed (debian/tmp/... if with extensions to be moved into -lib, or debian/python-X/... for pure Python module) version to make sure that everything got installed correctly. Also, if not too lengthy, I run unittests against all supported Python versions. Altogether, it does make things a bit more "cumbersome" but more reliable. Examples: http://git.debian.org/?p=pkg-exppsy/brian.git;a=blob;hb=HEAD;f=debian/rules https://github.com/neurodebian/dipy/blob/HEAD/debian/rules on a related note, more I hit it: - I set $HOME to point to build/, and at times MPLCONFIGDIR=$(HOME) where I echo "backend : Agg" >| $(MPLCONFIGDIR)/matplotlibrc to assure consistent matplotlib backend if graphics are built during documentation generation And indeed, written recommendations would be nice ;) On Tue, 22 Feb 2011, Andrey Rahmatullin wrote: > Hello. > Many Python modules ship unit tests which do not require module > instalation or user interaction and so can be run automatically during the > build time. They can be a very good QA tool, but it looks like they are > not used in all packages. > python-pylons: not used > python-pybabel: not used > pylint: used, but produce errors which are ignored > python-sphinx: apparently used > In many cases $PYTHON setup.py test (for all supported $PYTHON's) is > enough to run the tests and any errors mean that something is wrong with > the environment (provided the tests are correct). > Shouldn't we have some written recommendations (or even policy) on this > matter? -- =--= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110222135743.gx21...@onerussian.com
Re: Build-time testing
On Feb 22, 2011, at 03:51 PM, Andrey Rahmatullin wrote: >In many cases $PYTHON setup.py test (for all supported $PYTHON's) is >enough to run the tests and any errors mean that something is wrong with >the environment (provided the tests are correct). > >Shouldn't we have some written recommendations (or even policy) on this >matter? Probably so. Here's what I use in my flufl packages. Maybe there's a better way, but it works and runs the tests against all available Python2 interpreters. (I need to enable Python3 at some point.) -snip snip- #!/usr/bin/make -f PYTHON2=$(shell pyversions -vr) %: dh $@ --with python2, --buildsystem python_distutils test-python%: python$* setup.py test -vv override_dh_auto_test: $(PYTHON2:%=test-python%) -snip snip- It would be nice if dh auto-detected a setup.py (and/or missing Makefile) and didn't run 'make test' in that case, so that the override_dh_auto_test wasn't necessary. Yah, I should create a bug and patch for that. -Barry signature.asc Description: PGP signature
Re: Build-time testing
On 02/22/2011 03:47 PM, Barry Warsaw wrote: It would be nice if dh auto-detected a setup.py (and/or missing Makefile) and didn't run 'make test' in that case, so that the override_dh_auto_test wasn't necessary. Yah, I should create a bug and patch for that. I don;t think it would be a good idea to enable this as default due to the fun part of setuptools which starts to download random eggs from the internet when they're not available. My guess would eb that a lot of packages would miss that and we'd run into failures on the buildds. -- Bernd ZeimetzDebian GNU/Linux Developer http://bzed.dehttp://www.debian.org GPG Fingerprints: ECA1 E3F2 8E11 2432 D485 DD95 EB36 171A 6FF9 435F -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4d63cf1c.1020...@bzed.de
Re: Build-time testing
On Feb 22, 2011, at 03:58 PM, Bernd Zeimetz wrote: >On 02/22/2011 03:47 PM, Barry Warsaw wrote: > >> It would be nice if dh auto-detected a setup.py (and/or missing Makefile) >> and didn't run 'make test' in that case, so that the override_dh_auto_test >> wasn't necessary. Yah, I should create a bug and patch for that. > >I don;t think it would be a good idea to enable this as default due to the >fun part of setuptools which starts to download random eggs from the internet >when they're not available. My guess would eb that a lot of packages would >miss that and we'd run into failures on the buildds. You can disable that (and probably should), though I don't remember the exact options atm. -Barry signature.asc Description: PGP signature
Re: Build-time testing
On Tuesday, February 22, 2011 09:58:36 am Bernd Zeimetz wrote: > On 02/22/2011 03:47 PM, Barry Warsaw wrote: > > It would be nice if dh auto-detected a setup.py (and/or missing Makefile) > > and didn't run 'make test' in that case, so that the > > override_dh_auto_test wasn't necessary. Yah, I should create a bug and > > patch for that. > > I don;t think it would be a good idea to enable this as default due to > the fun part of setuptools which starts to download random eggs from the > internet when they're not available. My guess would eb that a lot of > packages would miss that and we'd run into failures on the buildds. Won't it only do that if the package isn't already installed? Scott K -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201102221117.48129.deb...@kitterman.com
Re: Build-time testing
On Feb 22, 2011, at 11:17 AM, Scott Kitterman wrote: >On Tuesday, February 22, 2011 09:58:36 am Bernd Zeimetz wrote: >> On 02/22/2011 03:47 PM, Barry Warsaw wrote: >> > It would be nice if dh auto-detected a setup.py (and/or missing Makefile) >> > and didn't run 'make test' in that case, so that the >> > override_dh_auto_test wasn't necessary. Yah, I should create a bug and >> > patch for that. >> >> I don;t think it would be a good idea to enable this as default due to >> the fun part of setuptools which starts to download random eggs from the >> internet when they're not available. My guess would eb that a lot of >> packages would miss that and we'd run into failures on the buildds. > >Won't it only do that if the package isn't already installed? I think that's right. I recently enabled the unittests in d/rules for computer-janitor and had to add the python-dbus and update-manager-core packages as B-D. Now that I'm thinking about it, the tests actually failed (in mysterious ways) until I added those B-Ds. But I don't think either package is in the Cheeseshop, and I don't remember if it tried to pull them and failed, or just didn't try. BTW, distribute doesn't print very helpful tracebacks when tests fail because of ImportErrors. The best way to debug that is by trying the test suite imports manually and then fixing the (better diagnosed) ImportErrors one-by-one. -B signature.asc Description: PGP signature
Re: Build-time testing
On Tue, Feb 22, 2011 at 11:17:47AM -0500, Scott Kitterman wrote: > > > It would be nice if dh auto-detected a setup.py (and/or missing Makefile) > > > and didn't run 'make test' in that case, so that the > > > override_dh_auto_test wasn't necessary. Yah, I should create a bug and > > > patch for that. > > > > I don;t think it would be a good idea to enable this as default due to > > the fun part of setuptools which starts to download random eggs from the > > internet when they're not available. My guess would eb that a lot of > > packages would miss that and we'd run into failures on the buildds. > > Won't it only do that if the package isn't already installed? Yes, it won't use the network if the required packages are found. -- WBR, wRAR signature.asc Description: Digital signature
Re: Build-time testing
On Tue, Feb 22, 2011 at 09:47:20AM -0500, Barry Warsaw wrote: > It would be nice if dh auto-detected a setup.py (and/or missing Makefile) and > didn't run 'make test' in that case, so that the override_dh_auto_test wasn't > necessary. Yah, I should create a bug and patch for that. Note that tests usually need more build-depends than the main packaging (which is in simple cases just a bunch of file operations), so enabling the tests for a package unfortunately involves manual expanding of build-depends. I would like to enable tests at least for packages I use, should I? -- WBR, wRAR signature.asc Description: Digital signature
Re: Build-time testing
OoO En cette fin de matinée radieuse du mardi 22 février 2011, vers 11:51, Andrey Rahmatullin disait : > Many Python modules ship unit tests which do not require module > instalation or user interaction and so can be run automatically during the > build time. They can be a very good QA tool, but it looks like they are > not used in all packages. > python-pylons: not used > python-pybabel: not used > pylint: used, but produce errors which are ignored > python-sphinx: apparently used > In many cases $PYTHON setup.py test (for all supported $PYTHON's) is > enough to run the tests and any errors mean that something is wrong with > the environment (provided the tests are correct). I wasn't aware of the existence of a test target. I usually use a special tailored script to run tests. See [1]. I switched to this kind of script because the use of PYTHONPATH could be misleading. I remember that despite the use of PYTHONPATH, some packages were tested against the installed version. I don't have this problem anymore since I now compile everything in a pbuilder but I kept the small script approach. [1] http://svn.debian.org/viewsvn/python-modules/packages/cerealizer/trunk/debian/runtests.py?view=markup > Shouldn't we have some written recommendations (or even policy) on this > matter? > -- > WBR, wRAR -- BOFH excuse #353: Second-sytem effect. pgpamUkWGQVDD.pgp Description: PGP signature
Re: Build-time testing
On Tue, Feb 22, 2011 at 07:23:51PM +0100, Vincent Bernat wrote: > I wasn't aware of the existence of a test target. It's a setuptools feature if I understand correctly (setuptools.command.test) and it needs specific test* arguments to setup(). Not all modules support it, some have custom runtests.sh scripts. -- WBR, wRAR signature.asc Description: Digital signature
Re: Build-time testing
On Feb 22, 2011, at 11:27 PM, Andrey Rahmatullin wrote: >On Tue, Feb 22, 2011 at 07:23:51PM +0100, Vincent Bernat wrote: >> I wasn't aware of the existence of a test target. >It's a setuptools feature if I understand correctly >(setuptools.command.test) and it needs specific test* arguments to >setup(). Not all modules support it, some have custom runtests.sh scripts. It's rumored to be part of setuptools, but of course we use distribute :). There it works very nicely, and even integrates with 2to3, including conversion of doctests. From my setup.py: -snip snip- setup( # ... test_suite='flufl.i18n.tests', # Auto-conversion to Python 3. use_2to3=True, convert_2to3_doctests=find_doctests(), use_2to3_fixers=['myfixers'], # ... -snip snip- find_doctests() is part of my setup_helpers package: -snip snip- def find_doctests(start='.', extension='.txt'): """Find separate-file doctests in the package. This is useful for Distribute's automatic 2to3 conversion support. The `setup()` keyword argument `convert_2to3_doctests` requires file names, which may be difficult to track automatically as you add new doctests. :param start: Directory to start searching in (default is cwd) :type start: string :param extension: Doctest file extension (default is .txt) :type extension: string :return: The doctest files found. :rtype: list """ doctests = [] for dirpath, dirnames, filenames in os.walk(start): doctests.extend(os.path.join(dirpath, filename) for filename in filenames if filename.endswith(extension)) return doctests -snip snip- I'd really love to see `python setup.py test` as the standard way to invoke a package's test suite. Cheers, -Barry signature.asc Description: PGP signature
Re: Build-time testing
On Feb 22, 2011, at 09:55 PM, Andrey Rahmatullin wrote: >On Tue, Feb 22, 2011 at 09:47:20AM -0500, Barry Warsaw wrote: >> It would be nice if dh auto-detected a setup.py (and/or missing Makefile) >> and didn't run 'make test' in that case, so that the override_dh_auto_test >> wasn't necessary. Yah, I should create a bug and patch for that. >Note that tests usually need more build-depends than the main packaging >(which is in simple cases just a bunch of file operations), so enabling >the tests for a package unfortunately involves manual expanding of >build-depends. Yep, but it's usually not so bad. See my previous message about computer-janitor. Note that it's usually the case that you're copying some Depends into Build-Depends. >I would like to enable tests at least for packages I use, should I? IMO, yes! Test are a Good Thing, running them regularly are Even Better. :) -Barry signature.asc Description: PGP signature
