Hi Martin,
I'm looking into de-duplicating reprotest's embedded copies of autopkgtest code.
In #833407 you said "Using the Python API (adt_testbed.py) is a lot easier and
more robust than talking to the backend servers directly", but adt_testbed.py
is currently not available as a public Python module. How do you expect users
to use it?
Might I suggest you make "/usr/share/autopkgtest/lib" available as a python
package? autopkgtest is a fine name for it. I understand you might have some
reservations about exposing some "internal" files like this, but the Pythonic
way is to prefix those with an underscore (as the stdlib does) and/or document
this fact clearly.
By contrast, putting a whole python package into a private path and then
hacking sys.path to be able to find this, is not really the appropriate
"Pythonic" way to do this. I can certainly sympathise, if you have
philosophical problems with Python's approach to module-hiding, but I think
this approach is not the best way to fight that fight. Rather, one would try to
make a PEP that constructs something clean, that everyone can use.
I've also attached the additional changes that Ceridwen made to the code, since
she imported it just before 4.0 was released. I have yet to study it in detail,
but feel free to comment in the meantime - if you see any major
incompatibilities that would be hard to fix in reprotest, it would be good for
me to know quickly and early.
X
On Sat, 3 Sep 2016 13:47:26 -0300 Antonio Terceiro <[email protected]> wrote:
> Package: reprotest
> Version: 0.2
> Severity: normal
>
> The virtualization backend binaries provided by autopkgtest are supposed
> to be consumable by external tools¹. Maybe we are not in a point where
> actually doing that is easy/convenient, but having a full copy of the
> autopkgtest code base in reprotest does not seem to be right.
>
> I even understand how this helped bootstrap the project and made it
> easier at the beginning, but I'm sure you will agree with me that reuse
> by copy-and-paste and embedded code copies are not good ideas.
>
> ¹ see e.g. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=833407
>
> -- System Information:
> Debian Release: stretch/sid
> APT prefers unstable-debug
> APT policy: (500, 'unstable-debug'), (500, 'testing-debug'), (500,
> 'unstable'), (500, 'testing'), (1, 'experimental-debug'), (1, 'experimental')
> Architecture: amd64 (x86_64)
>
> Kernel: Linux 4.6.0-1-amd64 (SMP w/4 CPU cores)
> Locale: LANG=pt_BR.UTF-8, LC_CTYPE=pt_BR.UTF-8 (charmap=UTF-8)
> Shell: /bin/sh linked to /bin/dash
> Init: systemd (via /run/systemd/system)
>
> Versions of packages reprotest depends on:
> ii apt-utils 1.3~rc4
> ii diffoscope 59
> ii libdpkg-perl 1.18.10
> ii procps 2:3.3.12-2
> ii python3-debian 0.1.29
> pn python3:any <none>
>
> Versions of packages reprotest recommends:
> ii autodep8 0.8
> ii disorderfs 0.4.3-1
> ii locales-all 2.24-1
> ii qemu-system 1:2.6+dfsg-3
> ii qemu-utils 1:2.6+dfsg-3
> ii schroot 1.6.10-2+b1
>
> reprotest suggests no packages.
>
> -- no debconf information
--
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git
diff --git a/reprotest/lib/VirtSubproc.py b/reprotest/lib/VirtSubproc.py
index 19b4e0f..c37051d 100644
--- a/reprotest/lib/VirtSubproc.py
+++ b/reprotest/lib/VirtSubproc.py
@@ -35,7 +35,7 @@ import pipes
import socket
import shutil
-import adtlog
+from reprotest.lib import adtlog
progname = "<VirtSubproc>"
devnull_read = open('/dev/null', 'r')
@@ -183,8 +183,8 @@ def check_exec(argv, downp=False, outp=False, timeout=0):
stdout=stdout, stderr=subprocess.PIPE)
if status:
- bomb("%s%s failed (exit status %d)" %
- ((downp and "(down) " or ""), argv, status))
+ bomb("%s%s failed (exit status %d)\n%s" %
+ ((downp and "(down) " or ""), argv, status, err))
if err:
bomb("%s unexpectedly produced stderr output `%s'" %
(argv, err))
diff --git a/reprotest/lib/__init__.py b/reprotest/lib/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/reprotest/lib/adt_testbed.py b/reprotest/lib/adt_testbed.py
index 7904bf6..7a55be3 100644
--- a/reprotest/lib/adt_testbed.py
+++ b/reprotest/lib/adt_testbed.py
@@ -33,10 +33,15 @@ import tempfile
import shutil
import urllib.parse
-from debian import debian_support
-import adtlog
-import VirtSubproc
+# TODO: removing this import disables install_tmp, may want to restore
+# it at some point if I'm improving support for building Debian packages in
+# particular.
+
+# from debian import debian_support
+
+from reprotest.lib import adtlog
+from reprotest.lib import VirtSubproc
timeouts = {'short': 100, 'copy': 300, 'install': 3000, 'test': 10000,
@@ -326,7 +331,7 @@ class Testbed:
def bomb(self, m, _type=adtlog.TestbedFailure):
adtlog.debug('%s %s' % (_type.__name__, m))
- self.stop()
+ # self.stop()
raise _type(m)
def badpkg(self, m):
@@ -384,6 +389,9 @@ class Testbed:
ll = list(map(urllib.parse.unquote, ll))
return ll
+ # TODO: with stdout and stderr defaulting to None, this function
+ # eats all errors/output from its call, which is not the right
+ # thing.
def execute(self, argv, xenv=[], stdout=None, stderr=None, kind='short'):
'''Run command in testbed.
@@ -408,6 +416,7 @@ class Testbed:
if env:
argv = ['env'] + env + argv
+ # import pdb; pdb.set_trace()
VirtSubproc.timeout_start(timeouts[kind])
try:
proc = subprocess.Popen(self.exec_cmd + argv,
@@ -437,11 +446,16 @@ class Testbed:
adtlog.debug('testbed command exited with code %i' % proc.returncode)
if proc.returncode in (254, 255):
- self.bomb('testbed auxverb failed with exit code %i' % proc.returncode)
+ msg = 'testbed auxverb failed with exit code %i' % proc.returncode
+ if out:
+ msg += '\n---- stdout ----\n%s----------------\n' % out
+ if err:
+ msg += '\n---- stderr ----\n%s----------------\n' % err
+ self.bomb(msg)
return (proc.returncode, out, err)
- def check_exec(self, argv, stdout=False, kind='short'):
+ def check_exec(self, argv, stdout=False, kind='short', xenv=[]):
'''Run argv in testbed.
If stdout is True, capture stdout and return it. Otherwise, don't
@@ -450,6 +464,7 @@ class Testbed:
argv must succeed and not print any stderr.
'''
(code, out, err) = self.execute(argv,
+ xenv=xenv,
stdout=(stdout and subprocess.PIPE or None),
stderr=subprocess.PIPE, kind=kind)
if err:
