Strange *.so names in pbuild do require tricks for test suite (Was: Trying to run build time tests using nosetest for python-pysam)
Hi Bas,
On Mon, Jul 20, 2015 at 11:06:22AM +0200, sebastic wrote:
> On 2015-07-20 10:54, Andreas Tille wrote:
> >unfortunately I did not got any response from Debian Python team and
> >I'm simply lost how to run the tests.
>
> Maybe change the dh_auto_test override to something like:
>
> override_dh_auto_test:
> PYBUILD_SYSTEM=custom \
> PYBUILD_TEST_ARGS="cd tests && {interpreter} pysam_test.py"
> dh_auto_test ||
> echo "Ignoring test failures"
>
> Also note the following in tests/00README.txt:
Seems this README is outdated - the file pysam_test.py is not provided
any more and the tests are run with nosetest now. However, I was able
to make some progress based from your input and I also patched the test
suite to work offline (by dropping two tests). The current rules file
is available in Git[1].
However, I have spotted a very strange problem I would like to discuss
with Debian Python team. The following dynamic libraries are created
in the build process:
$ find .pybuild -name "*.so"
.pybuild/pythonX.Y_3.4/build/pysam/csamfile.cpython-34m.so
.pybuild/pythonX.Y_3.4/build/pysam/libchtslib.cpython-34m.so
.pybuild/pythonX.Y_3.4/build/pysam/TabProxies.cpython-34m.so
.pybuild/pythonX.Y_3.4/build/pysam/calignmentfile.cpython-34m.so
.pybuild/pythonX.Y_3.4/build/pysam/csamtools.cpython-34m.so
.pybuild/pythonX.Y_3.4/build/pysam/cbcf.cpython-34m.so
.pybuild/pythonX.Y_3.4/build/pysam/ctabix.cpython-34m.so
.pybuild/pythonX.Y_3.4/build/pysam/cfaidx.cpython-34m.so
.pybuild/pythonX.Y_3.4/build/pysam/cvcf.cpython-34m.so
.pybuild/pythonX.Y_2.7/build/pysam/csamfile.so
.pybuild/pythonX.Y_2.7/build/pysam/cvcf.so
.pybuild/pythonX.Y_2.7/build/pysam/libchtslib.so
.pybuild/pythonX.Y_2.7/build/pysam/TabProxies.so
.pybuild/pythonX.Y_2.7/build/pysam/csamtools.so
.pybuild/pythonX.Y_2.7/build/pysam/cfaidx.so
.pybuild/pythonX.Y_2.7/build/pysam/calignmentfile.so
.pybuild/pythonX.Y_2.7/build/pysam/ctabix.so
.pybuild/pythonX.Y_2.7/build/pysam/cbcf.so
As you might have noticed the Python 3.4 files are named
*.cpython-34m.so
This leads to the fact that the nosetest3 call would fail with
the error
.pybuild/pythonX.Y_3.4/build/pysam/cfaidx.so
not found (actually all dynamic libraries are seeked without the
".cpython-34m" inside the name. I hacked around this inside the rules
file by providing symlinks and also noticed that I need to remove these
symlinks again since dh_python3 would otherwise stumble upon these.
I wonder what would be the proper way to deal with this.
I also learned that if I would run the test suite after the
*.debs are installed it has problems with the so names as well
that in both Python versions now are:
cfaidx.x86_64-linux-gnu.so
(when build on amd64). I suspect that the test suite itself might be
buggy since this is the usual naming for the dynamic libraries in Python
packages (at least when comparing with other packages).
Any hint is appreciated.
Kind regards
Andreas.
[1]
https://anonscm.debian.org/cgit/debian-med/python-pysam.git/tree/debian/rules
--
http://fam-tille.de
--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150721144400.gn17...@an3as.eu
Re: Strange *.so names in pbuild do require tricks for test suite (Was: Trying to run build time tests using nosetest for python-pysam)
> [1] > https://anonscm.debian.org/cgit/debian-med/python-pysam.git/tree/debian/rules just a quick note (.so file names look correct and I didn't check anything further): you can simplyfy debian/rulels by adding this line: export PYBUILD_TEST_NOSE=1 and moving `make -C pysam_data` outside PYBUILD_TEST_ARGS (i.e. just invoking it directly in debian/rules, it's not interpreter specific, right?) it will also set the right PYTHONPATH which might fix your problem -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150721152054.gm18...@sar0.p1otr.com
Re: Strange *.so names in pbuild do require tricks for test suite (Was: Trying to run build time tests using nosetest for python-pysam)
[Piotr Ożarowski, 2015-07-21]
> export PYBUILD_TEST_NOSE=1
if tests are not installed, add also:
export PYBUILD_TEST_ARGS={dir}/tests
--
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl www.griffith.cc www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150721152917.gn18...@sar0.p1otr.com
Re: Strange *.so names in pbuild do require tricks for test suite (Was: Trying to run build time tests using nosetest for python-pysam)
Hi Piotr,
thanks for your quick help!
On Tue, Jul 21, 2015 at 05:29:18PM +0200, Piotr Ożarowski wrote:
> [Piotr Ożarowski, 2015-07-21]
> > export PYBUILD_TEST_NOSE=1
>
> if tests are not installed, add also:
>
> export PYBUILD_TEST_ARGS={dir}/tests
While it looks a lot more elegant if I do
export PYBUILD_TEST_NOSE=1
export PYBUILD_TEST_ARGS={dir}/tests
...
override_dh_auto_test:
# Tests are stumbling upon missing ex1.bam file ...
ln -s pysam_data/ex1.bam tests
cd tests && make -C pysam_data
dh_auto_test
This ends in:
...
==
ERROR: testParsing (tabix_test.TestVCFFromVCF)
--
Traceback (most recent call last):
File
"/home/andreas/debian-maintain/repack/python-pysam/pysam-0.8.3/tests/tabix_test.py",
line 806, in setUp
TestVCF.setUp(self)
File
"/home/andreas/debian-maintain/repack/python-pysam/pysam-0.8.3/tests/tabix_test.py",
line 664, in setUp
shutil.copyfile(self.filename, self.tmpfilename)
File "/usr/lib/python2.7/shutil.py", line 82, in copyfile
with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: 'tabix_data/example.vcf40'
--
Ran 531 tests in 1.706s
FAILED (errors=468)
E: pybuild pybuild:256: test: plugin distutils failed with: exit code=1: cd
/home/andreas/debian-maintain/repack/python-pysam/pysam-0.8.3/.pybuild/pythonX.Y_2.7/build;
python2.7 -m nose --with-doctest {dir}/tests
dh_auto_test: pybuild --test -i python{version} -p 2.7 --dir . returned exit
code 13
tabix_data is a subdirectory of tests and seems to be ignored with the
method you proposed.
Thanks anyway
Andreas.
--
http://fam-tille.de
--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150721154628.gp17...@an3as.eu
pybuild and proxies -- could we make "prohibition" optional?
talking about this ;-) :
$> grep -A3 http_proxy pybuild
if 'http_proxy' not in env:
env['http_proxy'] = 'http://127.0.0.1:9/'
if 'https_proxy' not in env:
env['https_proxy'] = 'https://127.0.0.1:9/'
which is overall GREAT since now I can drop off all those manual http*_proxy
exports in debian/rules where I know that the application has nothing to do
with the network. BUT such settings forbid e.g. running any tests even if
those start/use local server specifically initiated for testing. This is not
forbidden AFAIK by the policy and the only concern is our buildd farm which
says that even local interface might not be available. Some
tools/libraries are smart enough to not try accessing proxy if http_proxy is
set but empty, but some don't.
So, long story short... would it be ok if I propose/commit a change like
diff --git a/pybuild b/pybuild
index d7bd35a..4edc175 100755
--- a/pybuild
+++ b/pybuild
@@ -50,10 +50,12 @@ def main(cfg):
env = environ.copy()
if 'LC_ALL' not in env:
env['LC_ALL'] = 'C.UTF-8'
-if 'http_proxy' not in env:
-env['http_proxy'] = 'http://127.0.0.1:9/'
-if 'https_proxy' not in env:
-env['https_proxy'] = 'https://127.0.0.1:9/'
+
+if environ.get('PYBUILD_FAKE_PROXY', '1') == '1':
+if 'http_proxy' not in env:
+env['http_proxy'] = 'http://127.0.0.1:9/'
+if 'https_proxy' not in env:
+env['https_proxy'] = 'https://127.0.0.1:9/'
if cfg.system:
certainty = 99
to enable testing when it is feasible to do it without external interactions
etc?
--
Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Research Scientist,Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik
--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150721191750.ga24...@onerussian.com
Re: pybuild and proxies -- could we make "prohibition" optional?
> So, long story short... would it be ok if I propose/commit a change like no need, if you want to disable it, just set https_proxy to empty string f.e. if you want to enable it only in tests: override_dh_auto_test: http_proxy='' dh_auto_test -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 signature.asc Description: Digital signature
Re: pybuild and proxies -- could we make "prohibition" optional?
On 21 July 2015 at 20:17, Yaroslav Halchenko wrote:
> talking about this ;-) :
>
> $> grep -A3 http_proxy pybuild
> if 'http_proxy' not in env:
> env['http_proxy'] = 'http://127.0.0.1:9/'
> if 'https_proxy' not in env:
> env['https_proxy'] = 'https://127.0.0.1:9/'
>
> which is overall GREAT since now I can drop off all those manual http*_proxy
> exports in debian/rules where I know that the application has nothing to do
> with the network. BUT such settings forbid e.g. running any tests even if
> those start/use local server specifically initiated for testing. This is not
> forbidden AFAIK by the policy and the only concern is our buildd farm which
> says that even local interface might not be available. Some
> tools/libraries are smart enough to not try accessing proxy if http_proxy is
> set but empty, but some don't.
>
> So, long story short... would it be ok if I propose/commit a change like
>
> diff --git a/pybuild b/pybuild
> index d7bd35a..4edc175 100755
> --- a/pybuild
> +++ b/pybuild
> @@ -50,10 +50,12 @@ def main(cfg):
> env = environ.copy()
> if 'LC_ALL' not in env:
> env['LC_ALL'] = 'C.UTF-8'
> -if 'http_proxy' not in env:
> -env['http_proxy'] = 'http://127.0.0.1:9/'
> -if 'https_proxy' not in env:
> -env['https_proxy'] = 'https://127.0.0.1:9/'
> +
> +if environ.get('PYBUILD_FAKE_PROXY', '1') == '1':
> +if 'http_proxy' not in env:
> +env['http_proxy'] = 'http://127.0.0.1:9/'
> +if 'https_proxy' not in env:
> +env['https_proxy'] = 'https://127.0.0.1:9/'
>
> if cfg.system:
> certainty = 99
>
> to enable testing when it is feasible to do it without external interactions
> etc?
In practice however we do allow accessing debian ftp archive and
localhost services, thus:
no_proxy=localhost,*.debian.org
would probably be better imho for the balance of "don't allow general
network access", "yet allow sensible networking test-suites to run".
--
Regards,
Dimitri.
--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive:
https://lists.debian.org/canbhlugfezfqrw4zjndr5rrqysdbye2epv1kkou85xnavwo...@mail.gmail.com
Re: pybuild and proxies -- could we make "prohibition" optional?
As I have mentioned some tools might not be smart about empty values in http_proxy, try e.g. git-annex under such settings On July 21, 2015 4:16:11 PM EDT, "Piotr Ożarowski" wrote: >> So, long story short... would it be ok if I propose/commit a change >like > >no need, if you want to disable it, just set https_proxy to empty >string > >f.e. if you want to enable it only in tests: > > override_dh_auto_test: >http_proxy='' dh_auto_test >-- >Piotr Ożarowski Debian GNU/Linux Developer >www.ozarowski.pl www.griffith.cc www.debian.org >GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 -- Sent from a phone which beats iPhone.
