Tags: patch Hello everyone,
attached is a debdiff with an upgrade to the latest upstream version 1.3.3, which supports python3, and a patch removing any python2 dependencies (which is basically only removing the python2 tests). The binary package is renamed to python3-versuchung and now depends on python3. Unfortunately, the upstream maintainer does not tag the releases anymore in github, but the new releases are on pypi: https://pypi.org/project/versuchung/ Since this is my first patch I submit, I'm unsure how to proceed. Is it usually the package maintainer (in this case Christoph Egger), who takes care of signing and uploading to sid? Thanks for helping me out here. lintian by the way complains about the standards version and the compat version... not sure if this is a big problem. Also, building seems to leave some artifacts (folder versuchung.egg-info and the file tests/reinit_types/DownstreamReinitTypesTest-3e32260cc0788418d241e6134893e7f2/metadata), not sure if this is intentional. When building again, these need to be removed in any case. Best Regards, Jann On Fri, 30 Aug 2019 07:48:18 +0000 Matthias Klose <d...@debian.org> wrote: > Package: src:python-versuchung > Version: 1.1-3 > Severity: normal > Tags: sid bullseye > User: debian-pyt...@lists.debian.org > Usertags: py2removal > > Python2 becomes end-of-live upstream, and Debian aims to remove > Python2 from the distribution, as discussed in > https://lists.debian.org/debian-python/2019/07/msg00080.html > > Your package either build-depends, depends on Python2, or uses Python2 > in the autopkg tests. Please stop using Python2, and fix this issue > by one of the following actions. > > - Convert your Package to Python3. This is the preferred option. In > case you are providing a Python module foo, please consider dropping > the python-foo package, and only build a python3-foo package. Please > don't drop Python2 modules, which still have reverse dependencies, > just document them. > > This is the preferred option. > > - If the package is dead upstream, cannot be converted or maintained > in Debian, it should be removed from the distribution. If the > package still has reverse dependencies, raise the severity to > "serious" and document the reverse dependencies with the BTS affects > command. If the package has no reverse dependencies, confirm that > the package can be removed, reassign this issue to ftp.debian.org, > make sure that the bug priority is set to normal and retitle the > issue to "RM: PKG -- removal triggered by the Python2 removal". > > - If the package has still many users (popcon >= 300), or is needed to > build another package which cannot be removed, document that by > adding the "py2keep" user tag (not replacing the py2remove tag), > using the debian-pyt...@lists.debian.org user. Also any > dependencies on an unversioned python package (python, python-dev) > must not be used, same with the python shebang. These have to be > replaced by python2/python2.7 dependencies and shebang. > > This is the least preferred option. > > If the conversion or removal needs action on another package first, > please document the blocking by using the BTS affects command, like > > affects <bug number of blocking py2removal bug> + src:python-versuchung > > If there is no py2removal bug for that reverse-dependency, please file > a bug on this package (similar to this bug report). > > If there are questions, please refer to the wiki page for the removal: > https://wiki.debian.org/Python/2Removal, or ask for help on IRC > #debian-python, or the debian-pyt...@lists.debian.org mailing list. > >
diff -Nru python-versuchung-1.1/debian/changelog python-versuchung-1.3.3/debian/changelog --- python-versuchung-1.1/debian/changelog 2014-11-02 23:29:07.000000000 +0100 +++ python-versuchung-1.3.3/debian/changelog 2020-05-03 10:27:25.000000000 +0200 @@ -1,3 +1,11 @@ +python-versuchung (1.3.3-1) UNRELEASED; urgency=medium + + * Non-Maintainer Upload + * Upgrade to version 1.3.3 + * Ported to python3 (Closes: #938248) + + -- Jann Haber <ja...@selfnet.de> Sun, 03 May 2020 10:27:25 +0200 + python-versuchung (1.1-3) unstable; urgency=medium * Add dependency on git, time for the tests (Closes: #767600) diff -Nru python-versuchung-1.1/debian/control python-versuchung-1.3.3/debian/control --- python-versuchung-1.1/debian/control 2014-11-02 23:28:24.000000000 +0100 +++ python-versuchung-1.3.3/debian/control 2020-05-03 10:27:25.000000000 +0200 @@ -4,21 +4,21 @@ Maintainer: Christoph Egger <christ...@debian.org> Build-Depends: debhelper (>= 9~), - python, + python3, dh-python, - python-setuptools, - python-sphinx, + python3-setuptools, + python3-sphinx, git, time Standards-Version: 3.9.6 Homepage: https://github.com/stettberger/versuchung -Package: python-versuchung +Package: python3-versuchung Architecture: all Depends: ${shlibs:Depends}, ${misc:Depends}, - ${python:Depends}, + ${python3:Depends}, ${sphinxdoc:Depends} Description: toolbox for reproducible research Versuchung is a toolbox for reproducible research. It automates the diff -Nru python-versuchung-1.1/debian/patches/no-tests-python2.patch python-versuchung-1.3.3/debian/patches/no-tests-python2.patch --- python-versuchung-1.1/debian/patches/no-tests-python2.patch 1970-01-01 01:00:00.000000000 +0100 +++ python-versuchung-1.3.3/debian/patches/no-tests-python2.patch 2020-05-03 10:27:25.000000000 +0200 @@ -0,0 +1,16 @@ +Skip running tests for python2 +--- a/tests/Makefile ++++ b/tests/Makefile +@@ -6,11 +6,7 @@ + check: $(TESTS) + + define test_cmd +-$(1): py2-$(1) py3-$(1) +- +-py2-$(1): FORCE +- @echo -n "Running test python2: $(1)..." +- @cd $(1); PYTHONPATH=$(PWD)/../src python2 test.py ++$(1): py3-$(1) + + py3-$(1): + @echo -n "Running test python3: $(1)..." diff -Nru python-versuchung-1.1/debian/patches/series python-versuchung-1.3.3/debian/patches/series --- python-versuchung-1.1/debian/patches/series 1970-01-01 01:00:00.000000000 +0100 +++ python-versuchung-1.3.3/debian/patches/series 2020-05-03 10:27:25.000000000 +0200 @@ -0,0 +1 @@ +no-tests-python2.patch diff -Nru python-versuchung-1.1/debian/rules python-versuchung-1.3.3/debian/rules --- python-versuchung-1.1/debian/rules 2014-10-25 17:08:16.000000000 +0200 +++ python-versuchung-1.3.3/debian/rules 2020-05-03 10:21:47.000000000 +0200 @@ -1,11 +1,11 @@ #!/usr/bin/make -f %: - dh $@ --with=python2,sphinxdoc --buildsystem=pybuild + dh $@ --with=python3,sphinxdoc --buildsystem=pybuild override_dh_auto_build: PYTHONPATH=. http_proxy='127.0.0.1:9' sphinx-build -N -bhtml doc/ build/html dh_auto_build override_dh_auto_test: - python setup.py test + python3 setup.py test diff -Nru python-versuchung-1.1/doc/conf.py python-versuchung-1.3.3/doc/conf.py --- python-versuchung-1.1/doc/conf.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/doc/conf.py 2018-08-22 17:23:14.000000000 +0200 @@ -57,7 +57,7 @@ p = Popen("git describe --always", stdout=PIPE, stderr=STDOUT, shell=True) (stdout, _) = p.communicate() if p.returncode == 0 and len(stdout) > 0: - version = stdout.rstrip() + version = stdout.rstrip().decode() else: version = "0.1" diff -Nru python-versuchung-1.1/doc/types/filesystem.rst python-versuchung-1.3.3/doc/types/filesystem.rst --- python-versuchung-1.1/doc/types/filesystem.rst 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/doc/types/filesystem.rst 2018-08-22 17:23:14.000000000 +0200 @@ -2,7 +2,11 @@ ********************* .. autoclass:: versuchung.files.File - :members: path,flush,copy_contents,value,write + :members: path,flush,copy_contents,value,write,make_executable + +.. autoclass:: versuchung.files.Executable + :members: path, execute .. autoclass:: versuchung.files.Directory - :members: path,value,new_file, mirror_directory + :members: path,value,new_file, new_directory, mirror_directory + diff -Nru python-versuchung-1.1/README.md python-versuchung-1.3.3/README.md --- python-versuchung-1.1/README.md 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/README.md 2018-08-22 17:23:14.000000000 +0200 @@ -24,7 +24,7 @@ Documentation ============= -For the documentation, please refer to https://vamos.informatik.uni-erlangen.de/versuchung-doc/ +For the documentation, please refer to https://versuchung.readthedocs.io Getting the latest version ========================== diff -Nru python-versuchung-1.1/setup.py python-versuchung-1.3.3/setup.py --- python-versuchung-1.1/setup.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/setup.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,21 +1,23 @@ #!/usr/bin/env python -from distutils.core import setup -from distutils.cmd import Command -from distutils.spawn import spawn +from __future__ import print_function + +from setuptools import setup, Command + +import sys try: from sphinx.setup_command import BuildDoc cmdclass = {'doc': BuildDoc} except: - print "No Sphinx installed (python-sphinx) so no documentation can be build" + print("No Sphinx installed (python-sphinx) so no documentation can be build") cmdclass = {} class TestCommand(Command): user_options = [] def run(self): - spawn(["make", "-C", "tests"], verbose = 1) + self.spawn(["make", "-C", "tests", "PYTHON=%s" % (sys.executable,)]) def initialize_options(self): pass @@ -25,13 +27,18 @@ cmdclass["test"] = TestCommand +with open("README.md", "r") as fh: + long_description = fh.read() + version_info = { 'name': 'versuchung', - 'version': '1.1', + 'version': '1.3.3', 'description': 'A toolbox for experiments', 'author': 'Christian Dietrich', 'author_email': 'stettber...@dokucode.de', 'url': 'http://github.de/stettberger/versuchung', + 'long_description': long_description, + 'long_description_content_type': "text/markdown", 'license': 'GPLv3', 'classifiers': [ 'Development Status :: 4 - Beta', @@ -39,7 +46,10 @@ 'Intended Audience :: Science/Research', 'License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)', 'Operating System :: OS Independent', - 'Programming Language :: Python :: 2.4' ], + 'Programming Language :: Python :: 2.6', + 'Programming Language :: Python :: 3.5' + ], + 'include_package_data': True, } diff -Nru python-versuchung-1.1/src/versuchung/archives.py python-versuchung-1.3.3/src/versuchung/archives.py --- python-versuchung-1.1/src/versuchung/archives.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/src/versuchung/archives.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,17 +1,18 @@ # This file is part of versuchung. -# +# # versuchung is free software: you can redistribute it and/or modify it under the # terms of the GNU General Public License as published by the Free Software # Foundation, either version 3 of the License, or (at your option) any later # version. -# +# # versuchung is distributed in the hope that it will be useful, but WITHOUT ANY # WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A # PARTICULAR PURPOSE. See the GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License along with # versuchung. If not, see <http://www.gnu.org/licenses/>. +from __future__ import print_function from versuchung.types import Type, InputParameter from versuchung.files import Directory, Directory_op_with, File @@ -19,6 +20,11 @@ import logging import os import sys +import gzip +try: + from StringIO import StringIO as BytesIO +except Exception: + from io import BytesIO class TarArchive(Type, InputParameter, Directory_op_with): """Can be used as: **input parameter** @@ -78,9 +84,11 @@ extract_mode = "" if "tar.gz" in fn or "tgz" in fn: - extract_mode = "x" + extract_mode = "z" if "tar.bz2" in fn or "bzip2" in fn: extract_mode = "j" + if "tar.xz" in fn or "txz" in fn: + extract_mode = "J" with self.tmp_directory as d: try: @@ -90,7 +98,7 @@ pass with Directory(self.name) as d2: dirname = os.path.abspath(".") - (out, ret) = shell("tar %szvf %s", extract_mode, fn) + (out, ret) = shell("tar %sxvf %s", extract_mode, fn) if ret != 0: raise RuntimeError("Extracting of %s failed" % fn) @@ -183,7 +191,7 @@ (lines, ret) = shell(cmd) if ret != 0 or lines == 0: - print "\n".join(lines) + print("\n".join(lines)) sys.exit(-1) self.__hash = lines[0].split("\t")[0] @@ -225,7 +233,7 @@ (lines, ret) = shell(cmd, *args) if ret != 0: - print "\n".join(lines) + print("\n".join(lines)) sys.exit(-1) if not self.__shallow: @@ -234,7 +242,7 @@ (lines, ret) = shell(cmd, *args) if ret != 0: - print "\n".join(lines) + print("\n".join(lines)) sys.exit(-1) @@ -254,42 +262,33 @@ class GzipFile(File): - def original_filename(self): - return self.__original_filename + def __init__(self, default_filename=""): + File.__init__(self, default_filename, binary=True) @property def path(self): - path = File.path.fget(self) - if self.parameter_type == "input" and not os.path.exists(path): - shell("gunzip < %s > %s", self.__original_filename, - path) - return path + """Decompress file into the temporary directory and return path to this location""" + assert self.tmp_directory is not None, \ + "Can gunzip file only as part of an active experiment" - @property - def value(self): path = File.path.fget(self) - if self.parameter_type == "input" and not os.path.exists(path): - shell("gunzip < %s > %s", self.__original_filename, - path) - return File.value.fget(self) - - @value.setter - def value(self, value): - File.value.fset(self, value) - - def before_experiment_run(self, parameter_type): - self.parameter_type = parameter_type - if parameter_type == "input": - self.__original_filename = File.path.fget(self) - self.subobjects["filename"] = File(self.__original_filename) - filename = self.name + "_" + os.path.basename(self.path.rstrip(".gz")) - self.set_path(self.tmp_directory.path, filename) - - File.before_experiment_run(self, parameter_type) - - def after_experiment_run(self, parameter_type): - File.after_experiment_run(self, parameter_type) - if parameter_type == "output": - shell("gzip -c %s > %s.1", self.path, self.path) - shell("mv %s.1 %s", self.path, self.path) - + base = os.path.basename(path.rstrip(".gz")) + filename = os.path.join(self.tmp_directory.path, + self.name + "_" + base) + + if not os.path.exists(filename): + shell("gunzip < %s > %s", path, filename) + + return filename + + def after_read(self, value): + x = BytesIO(value) + fd = gzip.GzipFile(fileobj=x) + return fd.read().decode() + + def before_write(self, value): + x = BytesIO() + fd = gzip.GzipFile(fileobj=x, mode="w") + fd.write(value.encode()) + fd.close() + return x.getvalue() diff -Nru python-versuchung-1.1/src/versuchung/database.py python-versuchung-1.3.3/src/versuchung/database.py --- python-versuchung-1.1/src/versuchung/database.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/src/versuchung/database.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,17 +1,19 @@ # This file is part of versuchung. -# +# # versuchung is free software: you can redistribute it and/or modify it under the # terms of the GNU General Public License as published by the Free Software # Foundation, either version 3 of the License, or (at your option) any later # version. -# +# # versuchung is distributed in the hope that it will be useful, but WITHOUT ANY # WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A # PARTICULAR PURPOSE. See the GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License along with # versuchung. If not, see <http://www.gnu.org/licenses/>. +from __future__ import print_function + from versuchung.types import Type, InputParameter, OutputParameter import logging import sqlite3 @@ -91,7 +93,7 @@ directory = self.tmp_directory.new_directory(self.name) path = os.path.join(directory.path, "my.cnf") logging.debug("MYSQL_HOME=%s", path) - with os.fdopen(os.open(path, os.O_WRONLY | os.O_CREAT, 0600), 'w') as handle: + with os.fdopen(os.open(path, os.O_WRONLY | os.O_CREAT, 0o600), 'w') as handle: handle.write("""[client] host=%s user=%s @@ -135,7 +137,7 @@ self.dynamic_experiment.experiment_identifier, str(self.dynamic_experiment.metadata)) - @property + @property def handle(self): """:return: handle -- MySQLdb database handle""" assert self.__database_connection @@ -153,7 +155,7 @@ self.__database_connection.commit() return c - def create_table(self, name, fields = [("key", "text"), ("value", "text")], + def create_table(self, name, fields = [("key", "text"), ("value", "text")], keys = None, conflict_strategy = None): """Creates a new table in the database. ``name`` is the name of newly created table. The ``fields`` are a list of @@ -261,7 +263,7 @@ """:return: string -- path to the sqlite database file""" return os.path.join(self.base_directory, self.__database_path) - @property + @property def handle(self): """:return: handle -- sqlite3 database handle""" assert self.__database_connection @@ -278,7 +280,7 @@ - def create_table(self, name, fields = [("key", "text"), ("value", "text")], + def create_table(self, name, fields = [("key", "text"), ("value", "text")], keys = None, conflict_strategy = "REPLACE"): """Creates a new table in the database. ``name`` is the name of newly created table. The ``fields`` are a list of @@ -326,7 +328,7 @@ is text. If it's a tuple the first entry is the name and the second its type:: >>> [("foo", "integer"), "barfoo"] - + This will result in two columns, one with type integer and one with type text. If a db is given this one is used instead of a default sqlite database named ``sqlite3.db`` @@ -366,7 +368,7 @@ return real_fields def before_experiment_run(self, parameter_type): - # Add database object as an + # Add database object as an self.subobjects["database"] = self.__db Type.before_experiment_run(self, parameter_type) @@ -483,7 +485,7 @@ class Database_SQlite_Merger: def log(self, msg, *args): if self.logging: - print "merger: " + (msg % args) + print("merger: " + (msg % args)) def __init__(self, target_path, source_paths = [], logging = True): self.target_path = target_path @@ -535,7 +537,7 @@ cur = self.target.cursor() TableDictrows = set() - + for name in self.tables: rows = set() headers = None @@ -555,7 +557,7 @@ if headers == ["experiment", "key", "value"]: TableDictrows.update(rows) - + cur.execute("CREATE TABLE IF NOT EXISTS TableDict (experiment text, key text, value text," "UNIQUE (key) ON CONFLICT REPLACE)") cur.executemany("INSERT INTO TableDict (experiment, key, value) values(?,?,?)", @@ -569,15 +571,14 @@ self.collect_and_create_tables(drop = not update) self.collect_data() self.target.close() - + if __name__ == '__main__': import sys if len(sys.argv) < 2: - print sys.argv[0] + " <target-database-file> [<source-db1> <source-db2> ...]" - print " -- merges different versuchung sqlite databases into a single one" + print(sys.argv[0] + " <target-database-file> [<source-db1> <source-db2> ...]") + print(" -- merges different versuchung sqlite databases into a single one") sys.exit(-1) merger = Database_SQlite_Merger(sys.argv[1], sys.argv[2:]) merger.merge() - diff -Nru python-versuchung-1.1/src/versuchung/execute.py python-versuchung-1.3.3/src/versuchung/execute.py --- python-versuchung-1.1/src/versuchung/execute.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/src/versuchung/execute.py 2018-08-22 17:23:14.000000000 +0200 @@ -17,7 +17,10 @@ import logging import os import resource -import thread +try: + import thread +except ImportError: + import _thread as thread import time import pipes from versuchung.tools import AdviceManager, Advice @@ -68,7 +71,7 @@ command = command % args logging.debug("executing: " + command) - p = Popen(command, stdout=PIPE, stderr=STDOUT, shell=True) + p = Popen(command, stdout=PIPE, stderr=STDOUT, shell=True, universal_newlines=True) stdout = "" while True: x = p.stdout.readline() diff -Nru python-versuchung-1.1/src/versuchung/experiment.py python-versuchung-1.3.3/src/versuchung/experiment.py --- python-versuchung-1.1/src/versuchung/experiment.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/src/versuchung/experiment.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,17 +1,18 @@ # This file is part of versuchung. -# +# # versuchung is free software: you can redistribute it and/or modify it under the # terms of the GNU General Public License as published by the Free Software # Foundation, either version 3 of the License, or (at your option) any later # version. -# +# # versuchung is distributed in the hope that it will be useful, but WITHOUT ANY # WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A # PARTICULAR PURPOSE. See the GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License along with # versuchung. If not, see <http://www.gnu.org/licenses/>. +from __future__ import print_function from optparse import OptionParser import datetime @@ -124,12 +125,29 @@ """ Type.__init__(self) InputParameter.__init__(self) - self.title = self.__class__.__name__ self.static_experiment = self + if default_experiment_instance and not os.path.exists(default_experiment_instance): + default_experiment_instance = None + self.__reinit__(default_experiment_instance) + + def __reinit__(self, experiment_path): + self.__experiment_instance = experiment_path + + if experiment_path: + if "/" in experiment_path: + self.__experiment_instance = os.path.basename(experiment_path) + + self.base_directory = os.path.join(os.curdir, experiment_path) + self.base_directory = os.path.realpath(self.base_directory) + assert os.path.exists(self.base_directory) - self.__experiment_instance = default_experiment_instance - self.__metadata = None + with open(os.path.join(experiment_path, "metadata")) as fd: + metadata = eval(fd.read()) + self.__metadata = metadata + else: + self.base_directory = None + self.__metadata = None # Copy input and output objects self.inputs = JavascriptStyleDictAccess(copy.deepcopy(self.__class__.inputs)) @@ -137,11 +155,7 @@ self.outputs = JavascriptStyleDictAccess(copy.deepcopy(self.__class__.outputs)) self.o = self.outputs - if default_experiment_instance != None: - self.base_directory = os.path.join(os.curdir, self.__experiment_instance) - self.base_directory = os.path.realpath(self.base_directory) - else: - self.base_directory = os.path.realpath(os.curdir) + self.subobjects.clear() # Sanity checking for input parameters. for (name, inp) in self.inputs.items(): @@ -149,26 +163,45 @@ if type(inp) == LambdaType: continue if not isinstance(inp, InputParameter): - print "%s cannot be used as an input parameter" % name + print("%s cannot be used as an input parameter" % name) sys.exit(-1) self.subobjects[name] = inp + for (name, outp) in self.outputs.items(): if not isinstance(outp, OutputParameter): - print "%s cannot be used as an output parameter" % name + print("%s cannot be used as an output parameter" % name) sys.exit(-1) self.subobjects[name] = outp + + # Reinit Children Attributes + if experiment_path: + for (name, inp) in self.inputs.items(): + if hasattr(inp, "__reinit__"): + try: + inp.__reinit__(metadata[name]) + except: + logging.debug('Cannot reinit field %s. Setting it to None', name) + self.inputs[name] = None + else: + # We cannot reinit this input from metadata. Therefore it is better to clear it. + logging.debug('Cannot reinit field %s. Setting it to None', name) + self.inputs[name] = None + + + def __setup_parser(self): self.__parser = OptionParser("%prog <options>") self.__parser.add_option('-d', '--base-dir', dest='base_dir', action='store', help="Directory which is used for storing the experiment data", default = ".") - self.__parser.add_option('-l', '--list', dest='do_list', action='store_true', - help="list all experiment results") + self.__parser.add_option('--dummy', dest='dummy_result', action='store_true', + help="Use dummy result directory", + default=False) self.__parser.add_option('-s', '--symlink', dest='do_symlink', action='store_true', help="symlink the result dir (as newest)") - self.__parser.add_option('-v', '--verbose', dest='verbose', action='count', + self.__parser.add_option('-v', '--verbose', dest='verbose', action='count', default=0, help="increase verbosity (specify multiple times for more)") for (name, inp) in self.inputs.items(): @@ -176,12 +209,6 @@ continue inp.inp_setup_cmdline_parser(self.__parser) - def __setup_tmp_directory(self): - """Creat temporary directory and assign it to every input and - output directories tmp_directory slots""" - # Create temp directory - self.tmp_directory = Directory(tempfile.mkdtemp()) - self.subobjects["tmp_directory"] = self.tmp_directory def execute(self, args = [], **kwargs): """Calling this method will execute the experiment @@ -205,24 +232,18 @@ >>> experiment.execute(input_parameter="foo") """ self.dynamic_experiment = self + self.startup_directory = os.path.abspath(os.curdir) + self.subobjects.update() # Set up the argument parsing self.__setup_parser() (opts, args) = self.__parser.parse_args(args) - os.chdir(opts.base_dir) setup_logging(opts.verbose) self.__opts = opts self.__args = args - if self.__opts.do_list: - for experiment in os.listdir(self.base_directory): - if experiment.startswith(self.title): - print "EXP", experiment - self.__do_list(self.__class__(experiment)) - return None - for key in kwargs: if not hasattr(opts, key): raise AttributeError("No argument called %s" % key) @@ -231,16 +252,22 @@ # Set up the experiment self.before_experiment_run("output") + # Goto the output directory + os.chdir(self.base_directory) + try: self.run() - except: + except RuntimeError as e: # Clean up the tmp directory if self.suspend_on_error: - print "tmp-dir: %s" % self.tmp_directory.path + print(str(e)) + print("tmp-dir: %s" % self.tmp_directory.path) self.suspend_python() logging.error("Removing tmp directory") shutil.rmtree(self.tmp_directory.path) - raise + raise e + finally: + os.chdir(self.startup_directory) # Tear down the experiment self.after_experiment_run("output") @@ -264,20 +291,6 @@ os.kill(os.getpid(), signal.SIGSTOP) - def __do_list(self, experiment, indent = 0): - with open(os.path.join(experiment.base_directory, "metadata")) as fd: - content = fd.read() - d = eval(content) - content = experiment.__experiment_instance + "\n" + content - print "+%s%s" % ("-" * indent, - content.strip().replace("\n", "\n|" + (" " * (indent+1)))) - for dirname in d.values(): - if type(dirname) != type(""): - continue - if os.path.exists(os.path.join(dirname, "metadata")) and \ - os.path.realpath(dirname) != os.path.realpath(experiment.base_directory): - self.__do_list(Experiment(dirname), indent + 3) - def before_experiment_run(self, parameter_type): # When experiment run as input, just run the normal input handlers if parameter_type == "input": @@ -304,7 +317,8 @@ self.subobjects.update() # Now set up the experiment tmp directory - self.__setup_tmp_directory() + self.tmp_directory = Directory(tempfile.mkdtemp()) + self.subobjects["tmp_directory"] = self.tmp_directory for obj in self.inputs.values(): obj.before_experiment_run("input") @@ -320,15 +334,20 @@ for name in self.inputs: metadata.update( self.inputs[name].inp_metadata() ) m = hashlib.md5() - m.update("version %s" % str(self.version)) + m.update(("version %s" % str(self.version)).encode()) calc_metadata = self.filter_metadata(metadata) for key in sorted(calc_metadata.keys()): - m.update(key + " " + str(calc_metadata[key])) + m.update((key + " " + str(calc_metadata[key])).encode()) self.__experiment_instance = "%s-%s" %(self.title, m.hexdigest()) - self.base_directory = os.path.join(os.curdir, self.__experiment_instance) + if self.__opts.dummy_result: + base = self.tmp_directory.path + else: + base = self.__opts.base_dir + self.base_directory = os.path.join(base, self.__experiment_instance) self.base_directory = os.path.realpath(self.base_directory) + if os.path.exists(self.base_directory): logging.info("Removing all files from existing output directory") for f in glob.glob(os.path.join(self.base_directory, '*')): @@ -355,6 +374,14 @@ self.__metadata = metadata + + def symlink_name(self): + """If -s is given, this function returns the name of the symlink + object that is created in the base directory. + + """ + return self.title + def after_experiment_run(self, parameter_type): if parameter_type == "output": @@ -371,9 +398,13 @@ shutil.rmtree(self.tmp_directory.path) - # Create a Symlink to the newsest result set - if self.__opts.do_symlink: - link = os.path.join(self.__opts.base_dir, self.title) + # Create a Symlink to the newsest result set, if it is not + # already removed (--dummy) + if self.__opts.do_symlink and os.path.exists(self.base_directory): + link = os.path.join(self.base_directory, + "..", + self.symlink_name()) + link = os.path.abspath(link) if os.path.islink(link): os.unlink(link) @@ -393,20 +424,25 @@ def inp_extract_cmdline_parser(self, opts, args): self.__experiment_instance = self.inp_parser_extract(opts, None) if not self.__experiment_instance: - print "Missing argument for %s" % self.title + print("Missing argument for %s" % self.title) raise ExperimentError # Resolve symlink relative to the current directory - self.__experiment_instance = os.path.realpath(self.__experiment_instance) - self.__experiment_instance = self.__experiment_instance[len(os.path.realpath(os.curdir))+1:] + path = os.path.realpath(self.__experiment_instance) + path = os.path.abspath(path) + self.__experiment_instance = os.path.basename(path) + + self.base_directory = path + assert os.path.exists(self.base_directory), \ + "Base Directory does not exist" - self.base_directory = os.path.join(os.curdir, self.__experiment_instance) - self.base_directory = os.path.realpath(self.base_directory) + self.__reinit__(path) for (name, outp) in self.outputs.items(): del self.subobjects[name] self.subobjects[name] = outp + def inp_metadata(self): return {self.name: self.__experiment_instance} @@ -470,8 +506,7 @@ return inp elif outp != None: return outp - + raise AttributeError("'%s' object has no attribute '%s'" %(\ self.__class__.__name__, name)) - diff -Nru python-versuchung-1.1/src/versuchung/files.py python-versuchung-1.3.3/src/versuchung/files.py --- python-versuchung-1.1/src/versuchung/files.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/src/versuchung/files.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,25 +1,29 @@ # This file is part of versuchung. -# +# # versuchung is free software: you can redistribute it and/or modify it under the # terms of the GNU General Public License as published by the Free Software # Foundation, either version 3 of the License, or (at your option) any later # version. -# +# # versuchung is distributed in the hope that it will be useful, but WITHOUT ANY # WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A # PARTICULAR PURPOSE. See the GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License along with # versuchung. If not, see <http://www.gnu.org/licenses/>. from versuchung.types import InputParameter, OutputParameter, Type -from versuchung.tools import before -from cStringIO import StringIO +import versuchung.archives +try: + from cStringIO import StringIO +except ImportError: + from io import StringIO import shutil import csv import os, stat import hashlib +import fnmatch class FilesystemObject(InputParameter, OutputParameter, Type): def __init__(self, default_name=""): @@ -45,14 +49,20 @@ if not self.__force_enclosing_directory: if self.parameter_type == "input": if self.static_experiment == self.dynamic_experiment: - self.__enclosing_directory = os.path.abspath(os.curdir) + self.__enclosing_directory = self.dynamic_experiment.startup_directory else: self.__enclosing_directory = self.static_experiment.base_directory elif self.parameter_type == "output": assert self.static_experiment == self.dynamic_experiment self.__enclosing_directory = self.dynamic_experiment.base_directory + elif self.static_experiment is not None: + self.__enclosing_directory = self.static_experiment.base_directory else: self.__enclosing_directory = os.path.abspath(os.curdir) + + if os.path.isabs(self.__object_name): + return self.__object_name + return os.path.join(self.__enclosing_directory, self.__object_name) @property @@ -83,17 +93,24 @@ disk before the experiment finishes. """ - def __init__(self, default_filename=""): + def __init__(self, default_filename="", binary=False): FilesystemObject.__init__(self, default_filename) self.__value = None + self.__binary = binary + if binary: + self.__binary_mode = "b" + else: + self.__binary_mode = "" + + @property def value(self): """This attribute can be read and written and represent the content of the specified file""" if not self.__value: try: - with open(self.path) as fd: + with open(self.original_path, "r" + self.__binary_mode) as fd: self.__value = self.after_read(fd.read()) except IOError: # File couldn't be read @@ -104,6 +121,10 @@ def value(self, value): self.__value = value + @property + def original_path(self): + return File.path.fget(self) + def write(self, content, append = False): """Similar to the :attr:`value` property. If the parameter `append` is `False`, then the property :attr:`value` is reset @@ -121,9 +142,9 @@ def flush(self): """Flush the cached content of the file to disk""" - if not self.__value: + if self.__value == None: return - with open(self.path, "w+") as fd: + with open(self.original_path, "w" + self.__binary_mode + "+") as fd: v = self.before_write(self.value) if v is None: v = "" @@ -140,8 +161,8 @@ def make_executable(self): """makes a file exectuable (chmod +x $file)""" - st = os.stat(self.path) - os.chmod(self.path, st.st_mode | stat.S_IEXEC) + st = os.stat(self.original_path) + os.chmod(self.original_path, st.st_mode | stat.S_IEXEC) def after_read(self, value): """To provide filtering of file contents in subclasses, overrwrite this method. @@ -188,7 +209,7 @@ raise NotImplementedError def inp_metadata(self): - return {self.name + "-md5": hashlib.md5(open(self.path).read()).hexdigest()} + return {self.name + "-md5": hashlib.md5(open(self.path, "rb").read()).hexdigest()} def execute(self, cmdline, *args): """Does start the executable with meth:`versuchung.execute.shell` and @@ -213,45 +234,62 @@ class Directory(FilesystemObject, Directory_op_with): """Can be used as: **input parameter** and **output parameter** - Represents the contents of directory. It can also be used with the - **with**-keyword to change the current working directory temporarily to this - directory:: + Represents the contents of directory. The filename_filter is a + glob/fnmatch expression to filter the directories content and to + ensure that no file is generated that does not fit this pattern. + An useful example of this is an output Directory that matches only + *.log files and is directly located in the result directory: + + outputs = { + "logs": Directory(".", filename_filter="*.log") + } + + It can also be used with the **with**-keyword to change the + current working directory temporarily to this directory:: with directory as dir: # Do something with adjusted current working directory print os.curdir + """ - def __init__(self, default_filename=""): + def __init__(self, default_filename="", filename_filter="*"): FilesystemObject.__init__(self, default_filename) Directory_op_with.__init__(self) + self.filename_filter = filename_filter self.__value = None self.__new_files = [] - def ___ensure_dir_exists(self): + def __ensure_dir_exists(self): if not os.path.exists(self.path): os.mkdir(self.path) - # Ensure dir exists DECORATOR - __ensure_dir_exists = before(___ensure_dir_exists) - @property def value(self): """:return: list -- directories and files in given directory""" if not self.__value: self.__value = os.listdir(self.path) + self.__value = [x for x in self.__value + if fnmatch.fnmatch(x, self.filename_filter)] return self.__value def __iter__(self): for name in self.value: p = os.path.join(self.path, name) + if name in self.subobjects: + yield self.subobjects[name] + continue + if os.path.isdir(p): d = Directory(name) d.set_path(self.path, p) self.subobjects[name] = d yield d else: - f = File(name) + if p.endswith(".gz"): + f = versuchung.archives.GzipFile(name) + else: + f = File(name) f.set_path(self.path, p) self.subobjects[name] = f yield f @@ -259,22 +297,32 @@ def before_experiment_run(self, parameter_type): FilesystemObject.before_experiment_run(self, parameter_type) if parameter_type == "output": - self.___ensure_dir_exists() + self.__ensure_dir_exists() - @__ensure_dir_exists - def new_file(self, name): + def new_file(self, name, compressed=False): """Generate a new :class:`~versuchung.files.File` in the directory. It will be flushed automatically if the experiment is over.""" - f = File(name) + if not fnmatch.fnmatch(name, self.filename_filter): + raise RuntimeError("Filename {} does not match filter {}".\ + format(name, self.filename_filter)) + self.__ensure_dir_exists() + if compressed: + f = versuchung.archives.GzipFile(name) + else: + f = File(name) f.set_path(self.path, name) + f.value = "" self.subobjects[name] = f return f - @__ensure_dir_exists def new_directory(self, name): """Generate a new :class:`~versuchung.files.Directory` in the directory. The directory <name> must not be present before""" + if not fnmatch.fnmatch(name, self.filename_filter): + raise RuntimeError("Filename {} does not match filter {}".\ + format(name, self.filename_filter)) + self.__ensure_dir_exists() f = Directory(name) f.set_path(self.path, name) os.mkdir(f.path) @@ -282,7 +330,6 @@ return f - @__ensure_dir_exists def mirror_directory(self, path, include_closure = None): """Copies the contents of the given directory to this directory. @@ -291,6 +338,8 @@ (absolute) path in the origin directory, if it is mirrored. If it is None, all files are included.""" + self.__ensure_dir_exists() + if not include_closure: include_closure = lambda arg: True @@ -365,4 +414,3 @@ if type(row) != list: raise TypeError("list of values required") self.value.append(row) - diff -Nru python-versuchung-1.1/src/versuchung/search.py python-versuchung-1.3.3/src/versuchung/search.py --- python-versuchung-1.1/src/versuchung/search.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/src/versuchung/search.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,20 +1,24 @@ # This file is part of versuchung. -# +# # versuchung is free software: you can redistribute it and/or modify it under the # terms of the GNU General Public License as published by the Free Software # Foundation, either version 3 of the License, or (at your option) any later # version. -# +# # versuchung is distributed in the hope that it will be useful, but WITHOUT ANY # WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A # PARTICULAR PURPOSE. See the GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License along with # versuchung. If not, see <http://www.gnu.org/licenses/>. +from __future__ import print_function + import os import logging +from versuchung.types import List + def search_experiment_results(experiment_type, directory, selector = None): """In large experiment setups it is hard to keep track of all result sets, which were produced. Therefore a search on the @@ -29,7 +33,7 @@ :func:`search_selector_metadata`. >>> search_experiment_results(MyExperiment, ".", lambda e: "home" in e.path) - [<MyExperiment object at 0xb74805ec>] + List(MyExperiment, <MyExperiment object at 0xb74805ec>]) """ # Name -> Path experiment_map = {} @@ -52,7 +56,8 @@ if exp.path not in [e.path for e in experiment_map.values()]: experiment_map[experiment_name] = exp - return experiment_map.values() + return List(experiment_type, experiment_map.values()) + def search_experiment(experiment_type, directory, selector = None): """Like :func:`search_experiment_results`, but returns only one @@ -109,8 +114,8 @@ import sys from versuchung.experiment import Experiment if len(sys.argv) != 4: - print "%s <experiment-type> <field> <data>" % sys.argv[0] + print("%s <experiment-type> <field> <data>" % sys.argv[0]) sys.exit(-1) Experiment.__name__ = sys.argv[1] for exp in search_experiment_results(Experiment, ".", {sys.argv[2]: sys.argv[3]}): - print exp.path + print(exp.path) diff -Nru python-versuchung-1.1/src/versuchung/tex.py python-versuchung-1.3.3/src/versuchung/tex.py --- python-versuchung-1.1/src/versuchung/tex.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/src/versuchung/tex.py 2018-08-22 17:23:14.000000000 +0200 @@ -12,6 +12,8 @@ # You should have received a copy of the GNU General Public License along with # versuchung. If not, see <http://www.gnu.org/licenses/>. +from __future__ import print_function + from versuchung.files import File import re import os @@ -158,4 +160,4 @@ if __name__ == '__main__': import sys - print PgfKeyDict(sys.argv[1]) + print(PgfKeyDict(sys.argv[1])) diff -Nru python-versuchung-1.1/src/versuchung/tools.py python-versuchung-1.3.3/src/versuchung/tools.py --- python-versuchung-1.1/src/versuchung/tools.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/src/versuchung/tools.py 2018-08-22 17:23:14.000000000 +0200 @@ -13,6 +13,8 @@ # versuchung. If not, see <http://www.gnu.org/licenses/>. import logging +import sys +from functools import wraps class JavascriptStyleDictAccess(dict): def __init__(self, d): @@ -42,33 +44,6 @@ logging.basicConfig(level=l) -def before(decorator_argument): - """Decorator for executing functions before other functions""" - def decorator(func): - def wrapped(self, *args, **kwargs): - # Late binding - inb4 = decorator_argument - if type(decorator_argument) == str: - inb4 = getattr(self, decorator_argument) - - if "func_code" in dir(inb4): - argcount = inb4.func_code.co_argcount - else: - raise RuntimeError("Invalid argument to decorator") - - if argcount == 1: - inb4(self) - elif argcount == 0: - inb4() - else: - raise RuntimeError("Unexpected parameter count") - - return func(self, *args, **kwargs) - wrapped.__doc__ = func.__doc__ - return wrapped - return decorator - - class Singleton(object): _instance = None def __new__(cls, *args, **kwargs): @@ -96,9 +71,9 @@ @staticmethod def advicable(func): """Decorator to mark a function as advicable""" - if not "func_name" in dir(func): + if not "__call__" in dir(func): raise ValueError("No function adviced") - full_name = "%s.%s" % (func.__module__, func.func_name) + full_name = "%s.%s" % (func.__module__, func.__name__) self = AdviceManager() @@ -157,13 +132,22 @@ if self.enabled: return # Hook only in if the methods are overwritten - if self.before.im_func != Advice.before.im_func: - am.before[self.method].append(self.before) - if self.around.im_func != Advice.around.im_func: - am.around[self.method].append(self.around) - if self.after.im_func != Advice.after.im_func: - am.after[self.method].append(self.after) - self.enabled = True + if sys.version_info[0] == 2: + if self.before.im_func != Advice.before.im_func: + am.before[self.method].append(self.before) + if self.around.im_func != Advice.around.im_func: + am.around[self.method].append(self.around) + if self.after.im_func != Advice.after.im_func: + am.after[self.method].append(self.after) + self.enabled = True + elif sys.version_info[0] == 3: + if self.before.__func__ != Advice.before: + am.before[self.method].append(self.before) + if self.around.__func__ != Advice.around: + am.around[self.method].append(self.around) + if self.after.__func__ != Advice.after: + am.after[self.method].append(self.after) + self.enabled = True def before(self, args, kwargs): return (args, kwargs) diff -Nru python-versuchung-1.1/src/versuchung/types.py python-versuchung-1.3.3/src/versuchung/types.py --- python-versuchung-1.1/src/versuchung/types.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/src/versuchung/types.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,36 +1,45 @@ # This file is part of versuchung. -# +# # versuchung is free software: you can redistribute it and/or modify it under the # terms of the GNU General Public License as published by the Free Software # Foundation, either version 3 of the License, or (at your option) any later # version. -# +# # versuchung is distributed in the hope that it will be useful, but WITHOUT ANY # WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A # PARTICULAR PURPOSE. See the GNU General Public License for more details. -# +# # You should have received a copy of the GNU General Public License along with # versuchung. If not, see <http://www.gnu.org/licenses/>. +from __future__ import print_function + import os import csv -from cStringIO import StringIO +try: + from cStringIO import StringIO +except ImportError: + from io import StringIO from optparse import OptionParser import copy +import glob class SubObjects(dict): def __init__(self, type_object): dict.__init__(self) self.parent = type_object + def __setitem__(self, key, value): assert not key in self or self[key] == value, "Duplicated object name: %s = %s" % (key, value) dict.__setitem__(self, key, value) value.parent_object = self.parent self.update() + def update(self): if not "parent" in dir(self) and len(self) > 0: - print "You probably used python multiprocessing, this might break horrible" + print("You probably used python multiprocessing, this might break horrible") return + for name, obj in self.items(): if self.parent.name != None: obj.name = "%s-%s" % (self.parent.name, name) @@ -59,6 +68,7 @@ def __init__(self): # We gather a list of objects that are used by us. self.subobjects = SubObjects(self) + self.__name = None def before_experiment_run(self, parameter_type): self.parameter_type = parameter_type @@ -112,9 +122,16 @@ assert self.dynamic_experiment, "Type is not used part of a running experiment" return self.dynamic_experiment.tmp_directory + def __repr__(self, value=None): + if value: + return "<%s %s '%s'>" %(self.__class__.__name__, self.__name, value) + return "<%s %s>" %(self.__class__.__name__, self.__name) + class InputParameter: + is_restartable = False + def __init__(self): pass def inp_setup_cmdline_parser(self, parser): @@ -181,8 +198,12 @@ Type.__init__(self) self.__value = default_value + def __reinit__(self, value): + self.__value = value + def inp_setup_cmdline_parser(self, parser): self.inp_parser_add(parser, None, self.__value) + def inp_extract_cmdline_parser(self, opts, args): self.__value = self.inp_parser_extract(opts, None) @@ -190,7 +211,10 @@ return {self.name: self.value} def __str__(self): - return self.value + return str(self.value) + + def __repr__(self): + return Type.__repr__(self, self.__value) @property def value(self): @@ -198,6 +222,7 @@ or the parameter given on the command line""" return self.__value + class Bool(InputParameter, Type): """Can be used as: **input parameter** @@ -208,6 +233,9 @@ Type.__init__(self) self.__value = default_value + def __reinit__(self, value): + self.__value = value + def inp_setup_cmdline_parser(self, parser): self.inp_parser_add(parser, None, self.__value) def inp_extract_cmdline_parser(self, opts, args): @@ -246,6 +274,10 @@ Type.__init__(self) self.__value = default_value + def __reinit__(self, value): + self.__value = value + + def inp_setup_cmdline_parser(self, parser): self.inp_parser_add(parser, None, self.__value) def inp_extract_cmdline_parser(self, opts, args): @@ -319,84 +351,97 @@ inherits from ``list``), so it is really easy to iterate over it:: for string in self.inputs.strings: - print string.value + print(string.value) for git in self.inputs.git: # Clone all given Git Archives - print git.path + print(git.path) """ def __init__(self, datatype, default_value=[]): InputParameter.__init__(self) Type.__init__(self) - list.__init__(self) - self.__default_value = default_value + list.__init__(self, default_value) if type(datatype) != type: datatype = type(datatype) self.datatype = datatype self.__command_line_parsed = False + def __reinit__(self, values): + if hasattr(self.datatype, "__reinit__"): + self[:] = [] + self.subobjects.clear() + for item in values: + # Intatiate Datatype + item = self.datatype(item) + self.subobjects["%d" % len(self)] = item + self.append(item) + + + def inp_setup_cmdline_parser(self, parser): - self.inp_parser_add(parser, None, copy.deepcopy(self.__default_value), action="append", + self.inp_parser_add(parser, None, [], action="append", help = "List parameter for type %s" % self.datatype.__name__) def before_experiment_run(self, parameter_type): - if parameter_type == "input" and \ - not self.__command_line_parsed: - count = 0 - for i in self.__default_value: - self.subobjects["%d" % count] = i - self.append(i) - count += 1 + for idx, value in enumerate(self): + self.subobjects[str(idx)] = value Type.before_experiment_run(self,parameter_type) def inp_extract_cmdline_parser(self, opts, args): import shlex args = self.inp_parser_extract(opts, None) - - # No argument where given, us the default_values in before_experiment_run - if len(args) == len(self.__default_value) and len(args) > 0\ - and type(args[0]) == type(args[0]) == self.datatype: + if not args: return - self.__command_line_parsed = True - - if len(args) > len(self.__default_value): - args = [x for x in args if type(x) != self.datatype] - - count = 0 - for arg in args: + # Remove default values + self[:] = [] + self.subobjects.clear() + + while len(args) > 0: + arg = args.pop(0) + if hasattr(self.datatype, "path") and not os.path.exists(arg): + args = glob.glob(arg) + args + # Remove duplicated items caused by symlinks + args = list(set([os.path.realpath(x) for x in args])) + continue # Create Subtype and initialize its parser subtype = self.datatype() - self.subobjects["%d" % count] = subtype - count += 1 + self.subobjects["%d" % len(self)] = subtype subtype_parser = OptionParser() subtype.inp_setup_cmdline_parser(subtype_parser) if not ":" in arg: - arg = ": " + arg + (opts, sub_args) = subtype_parser.parse_args(["--" + subtype.name, arg]) + else: + arg = arg.replace(": ", "--" + subtype.name + " ") + arg = arg.replace(":", "--" + subtype.name + "-") - arg = arg.replace(": ", "--" + subtype.name + " ") - arg = arg.replace(":", "--" + subtype.name + "-") + arg = shlex.split(arg) + (opts, sub_args) = subtype_parser.parse_args(arg) - arg = shlex.split(arg) + subtype.inp_extract_cmdline_parser(opts,sub_args) - (opts, args) = subtype_parser.parse_args(arg) - subtype.inp_extract_cmdline_parser(opts,args) self.append(subtype) + + def inp_metadata(self): - metadata = {} - for item in self: - metadata.update(item.inp_metadata()) + metadata = {self.name: []} + for idx, item in enumerate(self): + m = item.inp_metadata() + metadata[self.name].append(m["%s-%d" % (self.name, idx)]) + metadata.update(m) return metadata @property def value(self): """Returns the object (which behaves like a list) itself. This - is only implemented for a coherent API.""" + is only implemented for a coherent API.""" return self + def __repr__(self): + return Type.__repr__(self, list.__repr__(self)) diff -Nru python-versuchung-1.1/tests/advices/test.py python-versuchung-1.3.3/tests/advices/test.py --- python-versuchung-1.1/tests/advices/test.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/tests/advices/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,5 +1,7 @@ #!/usr/bin/python +from __future__ import print_function + from versuchung.experiment import Experiment from versuchung.files import File from versuchung.execute import shell @@ -23,5 +25,5 @@ if dirname: shutil.rmtree(dirname) - print "success" + print("success") diff -Nru python-versuchung-1.1/tests/basic_types/test.py python-versuchung-1.3.3/tests/basic_types/test.py --- python-versuchung-1.1/tests/basic_types/test.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/tests/basic_types/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,3 +1,5 @@ +from __future__ import print_function + from versuchung.experiment import Experiment from versuchung.types import String, Optional, Bool @@ -12,7 +14,7 @@ assert str(self.string) == "ABC" assert str(self.string) != repr(self.string) - assert "<versuchung.types.String" in repr(self.string) + assert "<String" in repr(self.string) assert "%s" % self.string == "ABC" assert self.bool.value == False @@ -22,5 +24,8 @@ import shutil t = BasicTypesTest() dirname = t(["--bool", "no"] + sys.argv) + + assert BasicTypesTest(dirname).bool.value == t.bool.value + shutil.rmtree(dirname) - print "success" + print("success") diff -Nru python-versuchung-1.1/tests/csv_read_write/test.py python-versuchung-1.3.3/tests/csv_read_write/test.py --- python-versuchung-1.1/tests/csv_read_write/test.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/tests/csv_read_write/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,3 +1,5 @@ +from __future__ import print_function + from versuchung.experiment import Experiment from versuchung.files import CSV_File @@ -19,4 +21,4 @@ if dirname: shutil.rmtree(dirname) - print "success" + print("success") diff -Nru python-versuchung-1.1/tests/database_basic/test.py python-versuchung-1.3.3/tests/database_basic/test.py --- python-versuchung-1.1/tests/database_basic/test.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/tests/database_basic/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,5 +1,7 @@ #!/usr/bin/python +from __future__ import print_function + from versuchung.experiment import Experiment from versuchung.database import Database, TableDict, Table import os @@ -37,12 +39,11 @@ assert os.path.exists(os.path.join(r1, "foobar.db")) e2 = SimpleExperiment2() - r2 = e2(se=r1) + r2 = e2(se=os.path.abspath(r1)) if r1: shutil.rmtree(r1) if r2: shutil.rmtree(r2) - print "success" - + print("success") diff -Nru python-versuchung-1.1/tests/database_mergetool/test.py python-versuchung-1.3.3/tests/database_mergetool/test.py --- python-versuchung-1.1/tests/database_mergetool/test.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/tests/database_mergetool/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,5 +1,7 @@ #!/usr/bin/python +from __future__ import print_function + from versuchung.experiment import Experiment from versuchung.types import Integer from versuchung.database import Database, TableDict, Table, Database_SQlite_Merger @@ -95,5 +97,5 @@ os.unlink("output.db") - print "success" + print("success") diff -Nru python-versuchung-1.1/tests/directory_output/test.py python-versuchung-1.3.3/tests/directory_output/test.py --- python-versuchung-1.1/tests/directory_output/test.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/tests/directory_output/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,11 +1,16 @@ +from __future__ import print_function + from versuchung.experiment import Experiment from versuchung.types import String -from versuchung.files import Directory +from versuchung.files import Directory, File +from versuchung.archives import GzipFile import os class SimpleExperiment(Experiment): outputs = {"dir1": Directory("d1"), - "dir2": Directory("d2")} + "dir2": Directory("d2"), + "filtered": Directory(".", filename_filter="*.log*"), + } def run(self): a = self.o.dir1.new_file("barfoo") @@ -18,6 +23,22 @@ self.o.dir2.mirror_directory(self.o.dir1.path, lambda x: True) + a = self.filtered.new_file("foo.log") + a.value = "xx" + try: + a = self.filtered.new_file("bar.xxx") + raise Exception("Filter does not work") + except RuntimeError as e: + pass # Everything is good + + b = self.filtered.new_file("barfoo.log.gz", compressed=True) + b.value = "xx" + + assert type(a) == File + assert type(b) == GzipFile + + + if __name__ == "__main__": import shutil, sys,os experiment = SimpleExperiment() @@ -27,6 +48,14 @@ assert os.path.exists(experiment.o.dir2.path + "/barfoo") assert os.path.exists(experiment.o.dir2.path + "/tmpdir/foo") + N = Directory(experiment.path, "*.log*") + assert experiment.filtered.value == N.value + assert os.path.exists(experiment.path + "/foo.log") + + contents = [x.value for x in N] + assert len(contents) == 2 + assert contents[0] == contents[1], contents + if dirname: shutil.rmtree(dirname) - print "success" + print("success") diff -Nru python-versuchung-1.1/tests/event_log/test.py python-versuchung-1.3.3/tests/event_log/test.py --- python-versuchung-1.1/tests/event_log/test.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/tests/event_log/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,3 +1,5 @@ +from __future__ import print_function + from versuchung.experiment import Experiment from versuchung.events import EventLog @@ -22,4 +24,4 @@ if dirname: shutil.rmtree(dirname) - print "success" + print("success") diff -Nru python-versuchung-1.1/tests/experiment_getattr/test.py python-versuchung-1.3.3/tests/experiment_getattr/test.py --- python-versuchung-1.1/tests/experiment_getattr/test.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/tests/experiment_getattr/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,5 +1,7 @@ #!/usr/bin/python +from __future__ import print_function + from versuchung.experiment import Experiment from versuchung.files import File from versuchung.types import String @@ -34,5 +36,5 @@ if dirname: shutil.rmtree(dirname) - print "success" + print("success") diff -Nru python-versuchung-1.1/tests/file_output/test.py python-versuchung-1.3.3/tests/file_output/test.py --- python-versuchung-1.1/tests/file_output/test.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/tests/file_output/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,11 +1,15 @@ +from __future__ import print_function + from versuchung.experiment import Experiment from versuchung.types import String -from versuchung.files import File +from versuchung.files import File, Directory +import os class SimpleExperiment(Experiment): inputs = {"input_key": String("default key"), "input_value": String("default value")} - outputs = {"output_file": File("output")} + outputs = {"output_file": File("output"), + "output_directory": Directory("output_directory")} def run(self): # Combine the input parameters @@ -14,13 +18,16 @@ # write the result to the output file self.outputs.output_file.value = content + "\n" - + # New output directory + x = self.output_directory.new_directory("foo").new_file("lala") if __name__ == "__main__": import shutil, sys experiment = SimpleExperiment() dirname = experiment(sys.argv) + assert os.path.exists("%s/output_directory/foo/lala" % dirname) + if dirname: shutil.rmtree(dirname) - print "success" + print("success") diff -Nru python-versuchung-1.1/tests/git_archive/test.py python-versuchung-1.3.3/tests/git_archive/test.py --- python-versuchung-1.1/tests/git_archive/test.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/tests/git_archive/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,11 +1,13 @@ #!/usr/bin/python +from __future__ import print_function + from versuchung.experiment import Experiment from versuchung.archives import TarArchive, GitArchive import os class GitArchiveTest(Experiment): - inputs = {"git": GitArchive(TarArchive("origin.tar.gz")), + inputs = {"git": GitArchive(TarArchive("origin.tar.gz")), "git_bare": GitArchive(TarArchive("origin.tar.gz"), shallow=True) } @@ -20,7 +22,7 @@ assert path == self.i.git.value.path assert os.path.abspath(os.curdir) == path - print "success" + print("success") if __name__ == "__main__": @@ -28,4 +30,11 @@ import shutil t = GitArchiveTest() dirname = t(sys.argv) + + # Reinit of Git Archive must fail + reinit = GitArchiveTest(dirname) + assert reinit.inputs['git'] is None + assert reinit.inputs['git_bare'] is None + + shutil.rmtree(dirname) diff -Nru python-versuchung-1.1/tests/gzip_file/test.py python-versuchung-1.3.3/tests/gzip_file/test.py --- python-versuchung-1.1/tests/gzip_file/test.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/tests/gzip_file/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,3 +1,5 @@ +from __future__ import print_function + from versuchung.experiment import Experiment from versuchung.archives import GzipFile from versuchung.files import Directory @@ -9,7 +11,7 @@ def run(self): with self.tmp_directory as d: assert self.tmp_directory.path in self.gz.path - assert self.gz.value.strip() == "CONTENT" + assert self.gz.value.strip() == "CONTENT", self.gz.value self.gz_out.value = "OUTPUT" @@ -19,8 +21,8 @@ experiment = SimpleExperiment() dirname = experiment(sys.argv) - assert len(open(experiment.gz_out.path).read()) > 0 + assert len(open(experiment.gz_out.original_path, 'rb').read()) > 0 if dirname: shutil.rmtree(dirname) - print "success" + print("success") diff -Nru python-versuchung-1.1/tests/invalid_experiment/test.py python-versuchung-1.3.3/tests/invalid_experiment/test.py --- python-versuchung-1.1/tests/invalid_experiment/test.py 1970-01-01 01:00:00.000000000 +0100 +++ python-versuchung-1.3.3/tests/invalid_experiment/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -0,0 +1,29 @@ +#!/usr/bin/python +from __future__ import print_function +import sys + +from versuchung.experiment import Experiment +from versuchung.types import List, String +from versuchung.files import File + +class OriginalExperiment(Experiment): + def run(self): + pass + +class TestExperiment(Experiment): + inputs = { 'experiment' : OriginalExperiment() } + + def run(self): + pass + + +if __name__ == "__main__": + experiment = TestExperiment() + + try: + dirname = experiment(["--experiment", "Invalid"]) + except: + print("success") + sys.exit(0) + + assert False diff -Nru python-versuchung-1.1/tests/list_parameter/test.py python-versuchung-1.3.3/tests/list_parameter/test.py --- python-versuchung-1.1/tests/list_parameter/test.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/tests/list_parameter/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,10 +1,15 @@ +from __future__ import print_function + from versuchung.experiment import Experiment from versuchung.types import String, List class SimpleExperiment(Experiment): inputs = {"strings": List(String(), default_value=[]), "default": List(String, default_value=[String("foo")]), - "default2": List(String, default_value=[String("foo")])} + "default2": List(String, default_value=[String("fox")]), + "default3": List(String, default_value=[String("a"), String("b")]) + } + def run(self): strings = [s.value for s in self.i.strings] @@ -27,4 +32,4 @@ if dirname: shutil.rmtree(dirname) - print "success" + print("success") diff -Nru python-versuchung-1.1/tests/list_parameter_lambda/test.py python-versuchung-1.3.3/tests/list_parameter_lambda/test.py --- python-versuchung-1.1/tests/list_parameter_lambda/test.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/tests/list_parameter_lambda/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,3 +1,5 @@ +from __future__ import print_function + from versuchung.experiment import Experiment from versuchung.types import String, List from versuchung.files import File @@ -57,4 +59,4 @@ for dirname in dirs_to_del: shutil.rmtree(dirname) - print "success" + print("success") diff -Nru python-versuchung-1.1/tests/machine_monitor/test.py python-versuchung-1.3.3/tests/machine_monitor/test.py --- python-versuchung-1.1/tests/machine_monitor/test.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/tests/machine_monitor/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,3 +1,5 @@ +from __future__ import print_function + from versuchung.experiment import Experiment from versuchung.execute import shell, MachineMonitor @@ -14,10 +16,10 @@ try: import psutil if not "phymem_usage" in dir(psutil): - print "skipped" + print("skipped") sys.exit(0) except: - print "skipped" + print("skipped") sys.exit(0) experiment = SimpleExperiment() @@ -25,4 +27,4 @@ if dirname: shutil.rmtree(dirname) - print "success" + print("success") diff -Nru python-versuchung-1.1/tests/Makefile python-versuchung-1.3.3/tests/Makefile --- python-versuchung-1.1/tests/Makefile 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/tests/Makefile 2018-08-22 17:23:14.000000000 +0200 @@ -1,19 +1,26 @@ TESTS = $(patsubst ./%,%,$(shell find . -maxdepth 1 -mindepth 1 -type d)) PHONY = $(TESTS) PWD = $(shell pwd) +PYTHON := python3 check: $(TESTS) define test_cmd -$(1): FORCE - @echo -n "Running test $(1)..." - @cd $(1); PYTHONPATH=$(PWD)/../src python test.py +$(1): py2-$(1) py3-$(1) + +py2-$(1): FORCE + @echo -n "Running test python2: $(1)..." + @cd $(1); PYTHONPATH=$(PWD)/../src python2 test.py + +py3-$(1): + @echo -n "Running test python3: $(1)..." + @cd $(1); PYTHONPATH=$(PWD)/../src python3 test.py endef $(foreach test,$(TESTS),$(eval $(call test_cmd,$(test)))) - + FORCE: .PHONY: $(PHONY) FORCE diff -Nru python-versuchung-1.1/tests/reinit_types/test.py python-versuchung-1.3.3/tests/reinit_types/test.py --- python-versuchung-1.1/tests/reinit_types/test.py 1970-01-01 01:00:00.000000000 +0100 +++ python-versuchung-1.3.3/tests/reinit_types/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -0,0 +1,51 @@ +from __future__ import print_function + +from versuchung.experiment import Experiment +from versuchung.types import String, Optional, Bool, List + +class ReinitTypesTest(Experiment): + inputs = {"string": String("A"), + "bool" : Bool(True), + "string_optional": Optional(String()), + 'list' : List(String, [])} + + def run(self): + assert self.string.value == "X" + assert self.bool.value == False + assert self.string_optional.value == None + + +class DownstreamReinitTypesTest(Experiment): + inputs = {'reinit': ReinitTypesTest('ReinitTypesTest-Foobar') } + + def run(self): + do_asserts(self.reinit) + +def do_asserts(reinit): + global t + + for field in ('string', 'bool', 'string_optional'): + assert getattr(t, field).value == getattr(reinit, field).value,\ + "Field %s not correctly reinitted" % field + + assert [x.value for x in t.list.value] == [x.value for x in reinit.list.value], \ + "Field list not correctly reinitted" + + +if __name__ == "__main__": + import sys + import shutil + t = ReinitTypesTest() + dirname = t(["--string", "X", "--bool", "no", "--list", "a", "--list", "b"]) + + # Reinit without enclosing experiment + reinit = ReinitTypesTest(dirname) + do_asserts(reinit) + + t2 = DownstreamReinitTypesTest() + dirname2 = t2(['--reinit', dirname]) + reinit2 = DownstreamReinitTypesTest(dirname2) + do_asserts(reinit2.reinit) + + shutil.rmtree(dirname) + print("success") diff -Nru python-versuchung-1.1/tests/run_shell/test.py python-versuchung-1.3.3/tests/run_shell/test.py --- python-versuchung-1.1/tests/run_shell/test.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/tests/run_shell/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,7 +1,12 @@ +from __future__ import print_function + from versuchung.experiment import Experiment from versuchung.execute import shell, shell_failok, CommandFailed import sys +import os + +experiment_file = os.path.abspath(__file__) class ShellExperiment(Experiment): def run(self): @@ -21,12 +26,12 @@ assert (['2 23'], 0) == shell("echo %(foo)s %(bar)s", {"foo": "2", "bar": "23"}) - shell("cat %s", __file__) + shell("cat %s", experiment_file) if __name__ == "__main__": import shutil experiment = ShellExperiment() dirname = experiment(sys.argv) - print "success" + print("success") if dirname: shutil.rmtree(dirname) diff -Nru python-versuchung-1.1/tests/search_experiments/test.py python-versuchung-1.3.3/tests/search_experiments/test.py --- python-versuchung-1.1/tests/search_experiments/test.py 1970-01-01 01:00:00.000000000 +0100 +++ python-versuchung-1.3.3/tests/search_experiments/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -0,0 +1,35 @@ +#!/usr/bin/python + +from versuchung.experiment import Experiment +from versuchung.types import * +from versuchung.files import * +from versuchung.files import Directory +from versuchung.execute import * +from versuchung.search import * +import sys + +class Exp1(Experiment): + inputs = {"in": String("hello")} + outputs = {"out": CSV_File("results.csv", delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)} + def run(self): + pass + +class Exp2(Experiment): + inputs = {"inp": lambda self: search_experiment_results(Exp1, ".", selector=None)} + outputs = {"out": CSV_File("results.csv", delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)} + def run(self): + assert len(self.inp) == 1 + +if __name__ == "__main__": + import sys + + exp1 = Exp1() + dirname1 = exp1(sys.argv) + + exp2 = Exp2() + dirname2 = exp2(sys.argv) + + import shutil + shutil.rmtree(dirname1) + shutil.rmtree(dirname2) + print("success") diff -Nru python-versuchung-1.1/tests/start_end_date/test.py python-versuchung-1.3.3/tests/start_end_date/test.py --- python-versuchung-1.1/tests/start_end_date/test.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/tests/start_end_date/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,3 +1,5 @@ +from __future__ import print_function + from versuchung.experiment import Experiment import time class SimpleExperiment(Experiment): @@ -13,4 +15,4 @@ if dirname: shutil.rmtree(dirname) - print "success" + print("success") diff -Nru python-versuchung-1.1/tests/string_list/test.py python-versuchung-1.3.3/tests/string_list/test.py --- python-versuchung-1.1/tests/string_list/test.py 1970-01-01 01:00:00.000000000 +0100 +++ python-versuchung-1.3.3/tests/string_list/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -0,0 +1,26 @@ +#!/usr/bin/python3 +from __future__ import print_function +import shutil + +from versuchung.experiment import Experiment +from versuchung.types import List, String +from versuchung.files import File + +class TestExperiment(Experiment): + inputs = { 'stringlist' : List(String) } + outputs = { 'result' : File("result") } + + def run(self): + for i in self.i.stringlist: + self.o.result.write(i.value) + + +if __name__ == "__main__": + experiment = TestExperiment() + + dirname = experiment(["--stringlist", "Hello world"]) + with open("%s/result" % dirname) as fd: + assert fd.read() == "Hello world" + + shutil.rmtree(dirname) + print("success") diff -Nru python-versuchung-1.1/tests/tar_archive/test.py python-versuchung-1.3.3/tests/tar_archive/test.py --- python-versuchung-1.1/tests/tar_archive/test.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/tests/tar_archive/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,5 +1,7 @@ #!/usr/bin/python +from __future__ import print_function + from versuchung.experiment import Experiment from versuchung.archives import TarArchive @@ -12,7 +14,7 @@ assert len(directory.value) == 2 assert "ABC" in directory.value assert "Hallo" in directory.value - print "success" + print("success") if __name__ == "__main__": diff -Nru python-versuchung-1.1/tests/tex_output/test.py python-versuchung-1.3.3/tests/tex_output/test.py --- python-versuchung-1.1/tests/tex_output/test.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/tests/tex_output/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,3 +1,5 @@ +from __future__ import print_function + from versuchung.experiment import Experiment from versuchung.tex import * @@ -40,4 +42,4 @@ assert dref["foobar"] == "42" shutil.rmtree(dirname) - print "success" + print("success") diff -Nru python-versuchung-1.1/tests/two_experiments/test.py python-versuchung-1.3.3/tests/two_experiments/test.py --- python-versuchung-1.1/tests/two_experiments/test.py 2014-10-25 16:48:48.000000000 +0200 +++ python-versuchung-1.3.3/tests/two_experiments/test.py 2018-08-22 17:23:14.000000000 +0200 @@ -1,3 +1,5 @@ +from __future__ import print_function + from versuchung.experiment import Experiment from versuchung.types import String from versuchung.files import File @@ -31,6 +33,9 @@ e1 = SimpleExperiment() r1 = e1([]) + assert SimpleExperiment(r1).output_file.path == e1.output_file.path,\ + "Default Constructor should set paths correct." + e2 = SimpleExperiment2() r2 = e2(se=r1) @@ -39,4 +44,4 @@ if r2: shutil.rmtree(r2) - print "success" + print("success")
