This tool aims to test U-Boot by executing U-Boot shell commands using the console interface. A single top-level script exists to execute or attach to the U-Boot console, run the entire script of tests against it, and summarize the results. Advantages of this approach are:
- Testing is performed in the same way a user or script would interact with U-Boot; there can be no disconnect. - There is no need to write or embed test-related code into U-Boot itself. It is asserted that writing test-related code in Python is simpler and more flexible that writing it all in C. - It is reasonably simple to interact with U-Boot in this way. A few simple tests are provided as examples. Soon, we should convert as many as possible of the other tests in test/* and test/cmd_ut.c too. In the future, I hope to publish (out-of-tree) the hook scripts, relay control utilities, and udev rules I will use for my own HW setup. See README.md for more details! Signed-off-by: Stephen Warren <swar...@wwwdotorg.org> Signed-off-by: Stephen Warren <swar...@nvidia.com> --- v2: Many fixes and tweaks have been squashed in. Separated out some of' the tests into separate commits, and added some more tests. --- test/py/.gitignore | 1 + test/py/README.md | 300 +++++++++++++++++++++++++++++++++++ test/py/conftest.py | 278 ++++++++++++++++++++++++++++++++ test/py/multiplexed_log.css | 76 +++++++++ test/py/multiplexed_log.py | 193 ++++++++++++++++++++++ test/py/pytest.ini | 9 ++ test/py/test.py | 24 +++ test/py/test_000_version.py | 13 ++ test/py/test_help.py | 6 + test/py/test_unknown_cmd.py | 8 + test/py/uboot_console_base.py | 185 +++++++++++++++++++++ test/py/uboot_console_exec_attach.py | 36 +++++ test/py/uboot_console_sandbox.py | 31 ++++ test/py/ubspawn.py | 97 +++++++++++ 14 files changed, 1257 insertions(+) create mode 100644 test/py/.gitignore create mode 100644 test/py/README.md create mode 100644 test/py/conftest.py create mode 100644 test/py/multiplexed_log.css create mode 100644 test/py/multiplexed_log.py create mode 100644 test/py/pytest.ini create mode 100755 test/py/test.py create mode 100644 test/py/test_000_version.py create mode 100644 test/py/test_help.py create mode 100644 test/py/test_unknown_cmd.py create mode 100644 test/py/uboot_console_base.py create mode 100644 test/py/uboot_console_exec_attach.py create mode 100644 test/py/uboot_console_sandbox.py create mode 100644 test/py/ubspawn.py diff --git a/test/py/.gitignore b/test/py/.gitignore new file mode 100644 index 000000000000..0d20b6487c61 --- /dev/null +++ b/test/py/.gitignore @@ -0,0 +1 @@ +*.pyc diff --git a/test/py/README.md b/test/py/README.md new file mode 100644 index 000000000000..23a403eb8d88 --- /dev/null +++ b/test/py/README.md @@ -0,0 +1,300 @@ +# U-Boot pytest suite + +## Introduction + +This tool aims to test U-Boot by executing U-Boot shell commands using the +console interface. A single top-level script exists to execute or attach to the +U-Boot console, run the entire script of tests against it, and summarize the +results. Advantages of this approach are: + +- Testing is performed in the same way a user or script would interact with + U-Boot; there can be no disconnect. +- There is no need to write or embed test-related code into U-Boot itself. + It is asserted that writing test-related code in Python is simpler and more + flexible that writing it all in C. +- It is reasonably simple to interact with U-Boot in this way. + +## Requirements + +The test suite is implemented using pytest. Interaction with the U-Boot console +involves executing some binary and interacting with its stdin/stdout. You will +need to implement various "hook" scripts that are called by the test suite at +the appropriate time. + +On Debian or Debian-like distributions, the following packages are required. +Similar package names should exist in other distributions. + +| Package | Version tested (Ubuntu 14.04) | +| -------------- | ----------------------------- | +| python | 2.7.5-5ubuntu3 | +| python-pytest | 2.5.1-1 | + +The test script supports either: + +- Executing a sandbox port of U-Boot on the local machine as a sub-process, + and interacting with it over stdin/stdout. +- Executing an external "hook" scripts to flash a U-Boot binary onto a + physical board, attach to the board's console stream, and reset the board. + Further details are described later. + +### Using `virtualenv` to provide requirements + +Older distributions (e.g. Ubuntu 10.04) may not provide all the required +packages, or may provide versions that are too old to run the test suite. One +can use the Python `virtualenv` script to locally install more up-to-date +versions of the required packages without interfering with the OS installation. +For example: + +```bash +$ cd /path/to/u-boot +$ sudo apt-get install python python-virtualenv +$ virtualenv venv +$ . ./venv/bin/activate +$ pip install pytest +``` + +## Testing sandbox + +To run the testsuite on the sandbox port (U-Boot built as a native user-space +application), simply execute: + +``` +./test/py/test.py --bd sandbox --build +``` + +The `--bd` option tells the test suite which board type is being tested. This +lets the test suite know which features the board has, and hence exactly what +can be tested. + +The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may +omit this option and build U-Boot yourself, in whatever way you choose, before +running the test script. + +The test script will attach to U-Boot, execute all valid tests for the board, +then print a summary of the test process. A complete log of the test session +will be written to `${build_dir}/test-log.html`. This is best viewed in a web +browser, but may be read directly as plain text, perhaps with the aid of the +`html2text` utility. + +## Command-line options + +- `--board-type`, `--bd`, `-B` set the type of the board to be tested. For + example, `sandbox` or `seaboard`. +- `--board-identity`, `--id` set the identity of the board to be tested. + This allows differentiation between multiple instances of the same type of + physical board that are attached to the same host machine. This parameter is + not interpreted by the test script in any way, but rather is simply passed + to the hook scripts described below, and may be used in any site-specific + way deemed necessary. +- `--build` indicates that the test script should compile U-Boot itself + before running the tests. If using this option, make sure that any + environment variables required by the build process are already set, such as + `$CROSS_COMPILE`. +- `--build-dir` sets the directory containing the compiled U-Boot binaries. + If omitted, this is `${source_dir}/build-${board_type}`. +- `--result-dir` sets the directory to write results, such as log files, + into. If omitted, the build directory is used. +- `--persistent-data-dir` sets the directory used to store persistent test + data. This is test data that may be re-used across test runs, such as file- + system images. + +`pytest` also implements a number of its own command-line options. Please see +`pytest` documentation for complete details. Execute `py.test --version` for +a brief summary. Note that U-Boot's test.py script passes all command-line +arguments directly to `pytest` for processing. + +## Testing real hardware + +The tools and techniques used to interact with real hardware will vary +radically between different host and target systems, and the whims of the user. +For this reason, the test suite does not attempt to directly interact with real +hardware in any way. Rather, it executes a standardized set of "hook" scripts +via `$PATH`. These scripts implement certain actions on behalf of the test +suite. This keeps the test suite simple and isolated from system variances +unrelated to U-Boot features. + +### Hook scripts + +#### Environment variables + +The following environment variables are set when running hook scripts: + +- `UBOOT_BOARD_TYPE` the board type being tested. +- `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was + specified. +- `UBOOT_SOURCE_DIR` the U-Boot source directory. +- `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory. +- `UBOOT_BUILD_DIR` the U-Boot build directory. +- `UBOOT_RESULT_DIR` the test result directory. +- `UBOOT_PERSISTENT_DATA_DIR` the test peristent data directory. + +#### `uboot-test-console` + +This script provides access to the U-Boot console. The script's stdin/stdout +should be connected to the board's console. This process should continue to run +indefinitely, until killed. The test suite will run this script in parallel +with all other hooks. + +This script may be implemented e.g. by exec()ing `cu`, `conmux`, etc. + +If you are able to run U-Boot under a hardware simulator such as qemu, then +you would likely spawn that simulator from this script. However, note that +`uboot-test-reset` may be called multiple times per test script run, and must +cause U-Boot to start execution from scratch each time. Hopefully your +simulator includes a virtual reset button! If not, you can launch the +simulator from `uboot-test-reset` instead, while arranging for this console +process to always communicate with the current simulator instance. + +#### `uboot-test-flash` + +Prior to running the test suite against a board, some arrangement must be made +so that the board executes the particular U-Boot binary to be tested. Often, +this involves writing the U-Boot binary to the board's flash ROM. The test +suite calls this hook script for that purpose. + +This script should perform the entire flashing process synchronously; the +script should only exit once flashing is complete, and a board reset will +cause the newly flashed U-Boot binary to be executed. + +It is conceivable that this script will do nothing. This might be useful in +the following cases: + +- Some other process has already written the desired U-Boot binary into the + board's flash prior to running the test suite. +- The board allows U-Boot to be downloaded directly into RAM, and executed + from there. Use of this feature will reduce wear on the board's flash, so + may be preferable if available, and if cold boot testing of U-Boot is not + required. If this feature is used, the `uboot-test-reset` script should + peform this download, since the board could conceivably be reset multiple + times in a single test run. + +It is up to the user to determine if those situations exist, and to code this +hook script appropriately. + +This script will typically be implemented by calling out to some SoC- or +board-specific vendor flashing utility. + +#### `uboot-test-reset` + +Whenever the test suite needs to reset the target board, this script is +executed. This is guaranteed to happen at least once, prior to executing the +first test function. If any test fails, the test infra-structure will execute +this script again to restore U-Boot to an operational state before running the +next test function. + +This script will likely be implemented by communicating with some form of +relay or electronic switch attached to the board's reset signal. + +The semantics of this script require that when it is executed, U-Boot will +start running from scratch. If the U-Boot binary to be tested has been written +to flash, pulsing the board's reset signal is likely all this script need do. +However, in some scenarios, this script may perform other actions. For +example, it may call out to some SoC- or board-specific vendor utility in order +to download the U-Boot binary directly into RAM and execute it. This would +avoid the need for `uboot-test-flash` to actually write U-Boot to flash, thus +saving wear on the flash chip(s). + +### Board-type-specific configuration + +Each board has a different configuration and behaviour. Many of these +differences can be automatically detected by parsing the `.config` file in the +build directory. However, some differences can't yet be handled automatically. + +For each board, an optional Python module `uboot_board_${board_type}` may exist +to provide board-specific information to the test script. Any global value +defined in these modules is available for use by any test function. The data +contained in these scripts must be purely derived from U-Boot source code. +Hence, these configuration files are part of the U-Boot source tree too. + +### Execution environment configuration + +Each user's hardware setup may enable testing different subsets of the features +implemented by a particular board's configuration of U-Boot. For example, a +U-Boot configuration may support USB device mode and USB Mass Storage, but this +can only be tested if a USB cable is connected between the board and the host +machine running the test script. + +For each board, optional Python modules `uboot_boardenv_${board_type}` and +`uboot_boardenv_${board_type}_${board_identity}` may exist to provide +board-specific and board-identity-specific information to the test script. Any +global value defined in these modules is available for use by any test +function. The data contained in these is specific to a particular user's +hardware configuration. Hence, these configuration files are not part of the +U-Boot source tree, and should be installed outside of the source tree. Users +should set `$PYTHONPATH` prior to running the test script to allow these +modules to be loaded. + +### Board module parameter usage + +The test scripts rely on the following variables being defined by the board +module: + +- None at present. + +### U-Boot `.config` feature usage + +The test scripts rely on various U-Boot `.config` features, either directly in +order to test those features, or indirectly in order to query information from +the running U-Boot instance in order to test other features. + +One example is that testing of the `md` command requires knowledge of a RAM +address to use for the test. This data is parsed from the output of the +`bdinfo` command, and hence relies on CONFIG_CMD_BDI being enabled. + +For a complete list of dependencies, please search the test scripts for +instances of: + +- `buildconfig.get(...` +- `@pytest.mark.buildconfigspec(...` + +### Complete invocation example + +Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and +any required environment configuration Python modules into $HOME/ubtest/py, +then you would likely invoke the test script as follows: + +If U-Boot has already been built: + +```bash +PATH=$HOME/ubtest/bin:$PATH \ + PYTHONPATH=${HOME}/ubtest/py:${PYTHONPATH} \ + ./test/py/test.py --bd seaboard +``` + +If you want the test script to compile U-Boot for you too, then you likely +need to set `$CROSS_COMPILE` to allow this, and invoke the test script as +follow: + +```bash +CROSS_COMPILE=arm-none-eabi- \ + PATH=$HOME/ubtest/bin:$PATH \ + PYTHONPATH=${HOME}/ubtest/py:${PYTHONPATH} \ + ./test/py/test.py --bd seaboard --build +``` + +## Writing tests + +Please refer to the pytest documentation for details of writing pytest tests. +Details specific to the U-Boot test suite are described below. + +A test fixture named `uboot_console` should be used by each test function. This +provides the means to interact with the U-Boot console, and retrieve board and +environment configuration information. + +The function `uboot_console.run_command()` executes a shell command on the +U-Boot console, and returns all output from that command. This allows +validation or interpretation of the command output. This function validates +that certain strings are not seen on the U-Boot console. These include shell +error messages and the U-Boot sign-on message (in order to detect unexpected +board resets). See the source of `uboot_console_base.py` for a complete list of +"bad" strings. Some test scenarios are expected to trigger these strings. Use +`uboot_console.disable_check()` to temporarily disable checking for specific +strings. See `test_unknown_cmd.py` for an example. + +Board- and board-environment configuration values may be accessed as sub-fields +of the `uboot_console.config` object, for example +`uboot_console.config.ram_base`. + +Build configuration values (from `.config`) may be accessed via the dictionary +`uboot_console.config.buildconfig`, with keys equal to the Kconfig variable +names. diff --git a/test/py/conftest.py b/test/py/conftest.py new file mode 100644 index 000000000000..b6efe03a60f8 --- /dev/null +++ b/test/py/conftest.py @@ -0,0 +1,278 @@ +# Copyright (c) 2015 Stephen Warren +# Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0 + +import atexit +import errno +import os +import os.path +import pexpect +import pytest +from _pytest.runner import runtestprotocol +import ConfigParser +import StringIO +import sys + +log = None +console = None + +def mkdir_p(path): + try: + os.makedirs(path) + except OSError as exc: + if exc.errno == errno.EEXIST and os.path.isdir(path): + pass + else: + raise + +def pytest_addoption(parser): + parser.addoption("--build-dir", default=None, + help="U-Boot build directory (O=)") + parser.addoption("--result-dir", default=None, + help="U-Boot test result/tmp directory") + parser.addoption("--persistent-data-dir", default=None, + help="U-Boot test persistent generated data directory") + parser.addoption("--board-type", "--bd", "-B", default="sandbox", + help="U-Boot board type") + parser.addoption("--board-identity", "--id", default="na", + help="U-Boot board identity/instance") + parser.addoption("--build", default=False, action="store_true", + help="Compile U-Boot before running tests") + +def pytest_configure(config): + global log + global console + global ubconfig + + test_py_dir = os.path.dirname(os.path.abspath(__file__)) + source_dir = os.path.dirname(os.path.dirname(test_py_dir)) + + board_type = config.getoption("board_type") + board_type_fn = board_type.replace("-", "_") + + board_identity = config.getoption("board_identity") + board_identity_fn = board_identity.replace("-", "_") + + build_dir = config.getoption("build_dir") + if not build_dir: + build_dir = source_dir + "/build-" + board_type + mkdir_p(build_dir) + + result_dir = config.getoption("result_dir") + if not result_dir: + result_dir = build_dir + mkdir_p(result_dir) + + persistent_data_dir = config.getoption("persistent_data_dir") + if not persistent_data_dir: + persistent_data_dir = build_dir + "/persistent-data" + mkdir_p(persistent_data_dir) + + import multiplexed_log + log = multiplexed_log.Logfile(result_dir + "/test-log.html") + + if config.getoption("build"): + if build_dir != source_dir: + o_opt = "O=%s" % build_dir + else: + o_opt = "" + cmds = ( + ["make", o_opt, "-s", board_type + "_defconfig"], + ["make", o_opt, "-s", "-j8"], + ) + runner = log.get_runner("make", sys.stdout) + for cmd in cmds: + runner.run(cmd, cwd=source_dir) + runner.close() + + class ArbitraryAttrContainer(object): + pass + + ubconfig = ArbitraryAttrContainer() + ubconfig.brd = dict() + ubconfig.env = dict() + + modules = [ + (ubconfig.brd, "uboot_board_" + board_type_fn), + (ubconfig.env, "uboot_boardenv_" + board_type_fn), + (ubconfig.env, "uboot_boardenv_" + board_type_fn + "_" + + board_identity_fn), + ] + for (sub_config, mod_name) in modules: + try: + mod = __import__(mod_name) + except ImportError: + continue + sub_config.update(mod.__dict__) + + ubconfig.buildconfig = dict() + + for conf_file in (".config", "include/autoconf.mk"): + dot_config = build_dir + "/" + conf_file + if not os.path.exists(dot_config): + raise Exception(conf_file + " does not exist; " + + "try passing --build option?") + + with open(dot_config, "rt") as f: + ini_str = "[root]\n" + f.read() + ini_sio = StringIO.StringIO(ini_str) + parser = ConfigParser.RawConfigParser() + parser.readfp(ini_sio) + ubconfig.buildconfig.update(parser.items("root")) + + ubconfig.test_py_dir = test_py_dir + ubconfig.source_dir = source_dir + ubconfig.build_dir = build_dir + ubconfig.result_dir = result_dir + ubconfig.persistent_data_dir = persistent_data_dir + ubconfig.board_type = board_type + ubconfig.board_identity = board_identity + + env_vars = ( + "board_type", + "board_identity", + "source_dir", + "test_py_dir", + "build_dir", + "result_dir", + "persistent_data_dir", + ) + for v in env_vars: + os.environ["UBOOT_" + v.upper()] = getattr(ubconfig, v) + + if board_type == "sandbox": + import uboot_console_sandbox + console = uboot_console_sandbox.ConsoleSandbox(log, ubconfig) + else: + import uboot_console_exec_attach + console = uboot_console_exec_attach.ConsoleExecAttach(log, ubconfig) + +def pytest_generate_tests(metafunc): + subconfigs = { + "brd": console.config.brd, + "env": console.config.env, + } + for fn in metafunc.fixturenames: + parts = fn.split("__") + if len(parts) < 2: + continue + if parts[0] not in subconfigs: + continue + subconfig = subconfigs[parts[0]] + vals = [] + val = subconfig.get(fn, []) + if val: + vals = (val, ) + else: + vals = subconfig.get(fn + "s", []) + metafunc.parametrize(fn, vals) + +@pytest.fixture(scope="session") +def uboot_console(request): + return console + +tests_not_run = set() +tests_failed = set() +tests_skipped = set() +tests_passed = set() + +def pytest_itemcollected(item): + tests_not_run.add(item.name) + +def cleanup(): + if console: + console.close() + if log: + log.status_pass("%d passed" % len(tests_passed)) + if tests_skipped: + log.status_skipped("%d skipped" % len(tests_skipped)) + for test in tests_skipped: + log.status_skipped("... " + test) + if tests_failed: + log.status_fail("%d failed" % len(tests_failed)) + for test in tests_failed: + log.status_fail("... " + test) + if tests_not_run: + log.status_fail("%d not run" % len(tests_not_run)) + for test in tests_not_run: + log.status_fail("... " + test) + log.close() +atexit.register(cleanup) + +def setup_boardspec(item): + mark = item.get_marker("boardspec") + if not mark: + return + required_boards = [] + for board in mark.args: + if board.startswith("!"): + if ubconfig.board_type == board[1:]: + pytest.skip("board not supported") + return + else: + required_boards.append(board) + if required_boards and ubconfig.board_type not in required_boards: + pytest.skip("board not supported") + +def setup_buildconfigspec(item): + mark = item.get_marker("buildconfigspec") + if not mark: + return + for option in mark.args: + if not ubconfig.buildconfig.get("config_" + option.lower(), None): + pytest.skip(".config feature not enabled") + +def pytest_runtest_setup(item): + log.start_section(item.name) + setup_boardspec(item) + setup_buildconfigspec(item) + +def pytest_runtest_protocol(item, nextitem): + reports = runtestprotocol(item, nextitem=nextitem) + failed = None + skipped = None + for report in reports: + if report.outcome == "failed": + failed = report + break + if report.outcome == "skipped": + if not skipped: + skipped = report + + if failed: + tests_failed.add(item.name) + elif skipped: + tests_skipped.add(item.name) + else: + tests_passed.add(item.name) + tests_not_run.remove(item.name) + + try: + if failed: + msg = "FAILED:\n" + str(failed.longrepr) + log.status_fail(msg) + elif skipped: + msg = "SKIPPED:\n" + str(skipped.longrepr) + log.status_skipped(msg) + else: + log.status_pass("OK") + except: + # If something went wrong with logging, it's better to let the test + # process continue, which may report other exceptions that triggered + # the logging issue (e.g. console.log wasn't created). Hence, just + # squash the exception. If the test setup failed due to e.g. syntax + # error somewhere else, this won't be seen. However, once that issue + # is fixed, if this exception still exists, it will then be logged as + # part of the test's stdout. + import traceback + print "Exception occurred while logging runtest status:" + traceback.print_exc() + # FIXME: Can we force a test failure here? + + log.end_section(item.name) + + if failed: + console.cleanup_spawn() + + return reports diff --git a/test/py/multiplexed_log.css b/test/py/multiplexed_log.css new file mode 100644 index 000000000000..96d87ebe034b --- /dev/null +++ b/test/py/multiplexed_log.css @@ -0,0 +1,76 @@ +/* + * Copyright (c) 2015 Stephen Warren + * + * SPDX-License-Identifier: GPL-2.0 + */ + +body { + background-color: black; + color: #ffffff; +} + +.implicit { + color: #808080; +} + +.section { + border-style: solid; + border-color: #303030; + border-width: 0px 0px 0px 5px; + padding-left: 5px +} + +.section-header { + background-color: #303030; + margin-left: -5px; + margin-top: 5px; +} + +.section-trailer { + display: none; +} + +.stream { + border-style: solid; + border-color: #303030; + border-width: 0px 0px 0px 5px; + padding-left: 5px +} + +.stream-header { + background-color: #303030; + margin-left: -5px; + margin-top: 5px; +} + +.stream-trailer { + display: none; +} + +.error { + color: #ff0000 +} + +.warning { + color: #ffff00 +} + +.info { + color: #808080 +} + +.action { + color: #8080ff +} + +.status-pass { + color: #00ff00 +} + +.status-skipped { + color: #ffff00 +} + +.status-fail { + color: #ff0000 +} diff --git a/test/py/multiplexed_log.py b/test/py/multiplexed_log.py new file mode 100644 index 000000000000..58b9a9c50ecf --- /dev/null +++ b/test/py/multiplexed_log.py @@ -0,0 +1,193 @@ +# Copyright (c) 2015 Stephen Warren +# Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0 + +import cgi +import os.path +import shutil +import subprocess + +mod_dir = os.path.dirname(os.path.abspath(__file__)) + +class LogfileStream(object): + def __init__(self, logfile, name, chained_file): + self.logfile = logfile + self.name = name + self.chained_file = chained_file + + def close(self): + pass + + def write(self, data, implicit=False): + self.logfile.write(self, data, implicit) + if self.chained_file: + self.chained_file.write(data) + + def flush(self): + self.logfile.flush() + if self.chained_file: + self.chained_file.flush() + +class RunAndLog(object): + def __init__(self, logfile, name, chained_file): + self.logfile = logfile + self.name = name + self.chained_file = chained_file + + def close(self): + pass + + def run(self, cmd, cwd=None): + msg = "+" + " ".join(cmd) + "\n" + if self.chained_file: + self.chained_file.write(msg) + self.logfile.write(self, msg) + + try: + p = subprocess.Popen(cmd, cwd=cwd, + stdin=None, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + (output, stderr) = p.communicate() + status = p.returncode + except subprocess.CalledProcessError as cpe: + output = cpe.output + status = cpe.returncode + self.logfile.write(self, output) + if status: + if self.chained_file: + self.chained_file.write(output) + raise Exception("command failed; exit code " + str(status)) + +class SectionCtxMgr(object): + def __init__(self, log, marker): + self.log = log + self.marker = marker + + def __enter__(self): + self.log.start_section(self.marker) + + def __exit__(self, extype, value, traceback): + self.log.end_section(self.marker) + +class Logfile(object): + def __init__(self, fn): + self.f = open(fn, "wt") + self.last_stream = None + self.linebreak = True + self.blocks = [] + self.cur_evt = 1 + shutil.copy(mod_dir + "/multiplexed_log.css", os.path.dirname(fn)) + self.f.write("""\ +<html> +<head> +<link rel="stylesheet" type="text/css" href="multiplexed_log.css"> +</head> +<body> +<tt> +""") + + def close(self): + self.f.write("""\ +</tt> +</body> +</html> +""") + self.f.close() + + def _escape(self, data): + data = data.replace(chr(13), "") + data = "".join((c in self._nonprint) and ("%%%02x" % ord(c)) or + c for c in data) + data = cgi.escape(data) + data = data.replace(" ", " ") + self.linebreak = data[-1:-1] == "\n" + data = data.replace(chr(10), "<br/>\n") + return data + + def _terminate_stream(self): + self.cur_evt += 1 + if not self.last_stream: + return + if not self.linebreak: + self.f.write("<br/>\n") + self.f.write("<div class=\"stream-trailer\" id=\"" + + self.last_stream.name + "\">End stream: " + + self.last_stream.name + "</div>\n") + self.f.write("</div>\n") + self.last_stream = None + + def _note(self, note_type, msg): + self._terminate_stream() + self.f.write("<div class=\"" + note_type + "\">\n") + self.f.write(self._escape(msg)) + self.f.write("<br/>\n") + self.f.write("</div>\n") + self.linebreak = True + + def start_section(self, marker): + self._terminate_stream() + self.blocks.append(marker) + blk_path = "/".join(self.blocks) + self.f.write("<div class=\"section\" id=\"" + blk_path + "\">\n") + self.f.write("<div class=\"section-header\" id=\"" + blk_path + + "\">Section: " + blk_path + "</div>\n") + + def end_section(self, marker): + if (not self.blocks) or (marker != self.blocks[-1]): + raise Exception("Block nesting mismatch: \"%s\" \"%s\"" % + (marker, "/".join(self.blocks))) + self._terminate_stream() + blk_path = "/".join(self.blocks) + self.f.write("<div class=\"section-trailer\" id=\"section-trailer-" + + blk_path + "\">End section: " + blk_path + "</div>\n") + self.f.write("</div>\n") + self.blocks.pop() + + def section(self, marker): + return SectionCtxMgr(self, marker) + + def error(self, msg): + self._note("error", msg) + + def warning(self, msg): + self._note("warning", msg) + + def info(self, msg): + self._note("info", msg) + + def action(self, msg): + self._note("action", msg) + + def status_pass(self, msg): + self._note("status-pass", msg) + + def status_skipped(self, msg): + self._note("status-skipped", msg) + + def status_fail(self, msg): + self._note("status-fail", msg) + + def get_stream(self, name, chained_file=None): + return LogfileStream(self, name, chained_file) + + def get_runner(self, name, chained_file=None): + return RunAndLog(self, name, chained_file) + + _nonprint = ("^%" + "".join(chr(c) for c in range(0, 32) if c != 10) + + "".join(chr(c) for c in range(127, 256))) + + def write(self, stream, data, implicit=False): + if stream != self.last_stream: + self._terminate_stream() + self.f.write("<div class=\"stream\" id=\"%s\">\n" % stream.name) + self.f.write("<div class=\"stream-header\" id=\"" + stream.name + + "\">Stream: " + stream.name + "</div>\n") + if implicit: + self.f.write("<span class=\"implicit\">") + self.f.write(self._escape(data)) + if implicit: + self.f.write("</span>") + self.last_stream = stream + + def flush(self): + self.f.flush() diff --git a/test/py/pytest.ini b/test/py/pytest.ini new file mode 100644 index 000000000000..1bdff810d36e --- /dev/null +++ b/test/py/pytest.ini @@ -0,0 +1,9 @@ +# Copyright (c) 2015 Stephen Warren +# Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0 + +[pytest] +markers = + boardspec: U-Boot: Describes the set of boards a test can/can't run on. + buildconfigspec: U-Boot: Describes Kconfig/config-header constraints. diff --git a/test/py/test.py b/test/py/test.py new file mode 100755 index 000000000000..7768216a2335 --- /dev/null +++ b/test/py/test.py @@ -0,0 +1,24 @@ +#!/usr/bin/env python + +# Copyright (c) 2015 Stephen Warren +# Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0 + +import os +import os.path +import sys + +sys.argv.pop(0) + +args = ["py.test", os.path.dirname(__file__)] +args.extend(sys.argv) + +try: + os.execvp("py.test", args) +except: + import traceback + traceback.print_exc() + print >>sys.stderr, """ +exec(py.test) failed; perhaps you are missing some dependencies? +See test/md/README.md for the list.""" diff --git a/test/py/test_000_version.py b/test/py/test_000_version.py new file mode 100644 index 000000000000..360c8fd726e0 --- /dev/null +++ b/test/py/test_000_version.py @@ -0,0 +1,13 @@ +# Copyright (c) 2015 Stephen Warren +# +# SPDX-License-Identifier: GPL-2.0 + +# pytest runs tests the order of their module path, which is related to the +# filename containing the test. This file is named such that it is sorted +# first, simply as a very basic sanity check of the functionality of the U-Boot +# command prompt. + +def test_version(uboot_console): + with uboot_console.disable_check("main_signon"): + response = uboot_console.run_command("version") + uboot_console.validate_main_signon_in_text(response) diff --git a/test/py/test_help.py b/test/py/test_help.py new file mode 100644 index 000000000000..3cc896ee7af8 --- /dev/null +++ b/test/py/test_help.py @@ -0,0 +1,6 @@ +# Copyright (c) 2015 Stephen Warren +# +# SPDX-License-Identifier: GPL-2.0 + +def test_help(uboot_console): + uboot_console.run_command("help") diff --git a/test/py/test_unknown_cmd.py b/test/py/test_unknown_cmd.py new file mode 100644 index 000000000000..ba12de56a294 --- /dev/null +++ b/test/py/test_unknown_cmd.py @@ -0,0 +1,8 @@ +# Copyright (c) 2015 Stephen Warren +# +# SPDX-License-Identifier: GPL-2.0 + +def test_unknown_command(uboot_console): + with uboot_console.disable_check("unknown_command"): + response = uboot_console.run_command("non_existent_cmd") + assert("Unknown command 'non_existent_cmd' - try 'help'" in response) diff --git a/test/py/uboot_console_base.py b/test/py/uboot_console_base.py new file mode 100644 index 000000000000..9f13fead2e7e --- /dev/null +++ b/test/py/uboot_console_base.py @@ -0,0 +1,185 @@ +# Copyright (c) 2015 Stephen Warren +# Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0 + +import multiplexed_log +import os +import pytest +import re +import sys + +pattern_uboot_spl_signon = re.compile("(U-Boot SPL \\d{4}\\.\\d{2}-[^\r\n]*)") +pattern_uboot_main_signon = re.compile("(U-Boot \\d{4}\\.\\d{2}-[^\r\n]*)") +pattern_stop_autoboot_prompt = re.compile("Hit any key to stop autoboot: ") +pattern_unknown_command = re.compile("Unknown command '.*' - try 'help'") +pattern_error_notification = re.compile("## Error: ") + +class ConsoleDisableCheck(object): + def __init__(self, console, check_type): + self.console = console + self.check_type = check_type + + def __enter__(self): + self.console.disable_check_count[self.check_type] += 1 + + def __exit__(self, extype, value, traceback): + self.console.disable_check_count[self.check_type] -= 1 + +class ConsoleBase(object): + def __init__(self, log, config, max_fifo_fill): + self.log = log + self.config = config + self.max_fifo_fill = max_fifo_fill + + self.logstream = self.log.get_stream("console", sys.stdout) + + # Array slice removes leading/trailing quotes + self.prompt = self.config.buildconfig["config_sys_prompt"][1:-1] + self.prompt_escaped = re.escape(self.prompt) + self.p = None + self.disable_check_count = { + "spl_signon": 0, + "main_signon": 0, + "unknown_command": 0, + "error_notification": 0, + } + + self.at_prompt = False + self.at_prompt_logevt = None + self.ram_base = None + + def close(self): + if self.p: + self.p.close() + self.logstream.close() + + def run_command(self, cmd, wait_for_echo=True, send_nl=True, wait_for_prompt=True): + self.ensure_spawned() + + if self.at_prompt and \ + self.at_prompt_logevt != self.logstream.logfile.cur_evt: + self.logstream.write(self.prompt, implicit=True) + + bad_patterns = [] + bad_pattern_ids = [] + if (self.disable_check_count["spl_signon"] == 0 and + self.uboot_spl_signon): + bad_patterns.append(self.uboot_spl_signon_escaped) + bad_pattern_ids.append("SPL signon") + if self.disable_check_count["main_signon"] == 0: + bad_patterns.append(self.uboot_main_signon_escaped) + bad_pattern_ids.append("U-Boot main signon") + if self.disable_check_count["unknown_command"] == 0: + bad_patterns.append(pattern_unknown_command) + bad_pattern_ids.append("Unknown command") + if self.disable_check_count["error_notification"] == 0: + bad_patterns.append(pattern_error_notification) + bad_pattern_ids.append("Error notification") + try: + self.at_prompt = False + if send_nl: + cmd += "\n" + while cmd: + # Limit max outstanding data, so UART FIFOs don't overflow + chunk = cmd[:self.max_fifo_fill] + cmd = cmd[self.max_fifo_fill:] + self.p.send(chunk) + if not wait_for_echo: + continue + chunk = re.escape(chunk) + chunk = chunk.replace("\\\n", "[\r\n]") + m = self.p.expect([chunk] + bad_patterns) + if m != 0: + self.at_prompt = False + raise Exception("Bad pattern found on console: " + + bad_pattern_ids[m - 1]) + if not wait_for_prompt: + return + m = self.p.expect([self.prompt_escaped] + bad_patterns) + if m != 0: + self.at_prompt = False + raise Exception("Bad pattern found on console: " + + bad_pattern_ids[m - 1]) + self.at_prompt = True + self.at_prompt_logevt = self.logstream.logfile.cur_evt + # Only strip \r\n; space/TAB might be significant if testing + # indentation. + return self.p.before.strip("\r\n") + except Exception as ex: + self.log.error(str(ex)) + self.cleanup_spawn() + raise + + def ctrlc(self): + self.run_command(chr(3), wait_for_echo=False, send_nl=False) + + def ensure_spawned(self): + if self.p: + return + try: + self.at_prompt = False + self.log.action("Starting U-Boot") + self.p = self.get_spawn() + # Real targets can take a long time to scroll large amounts of + # text if LCD is enabled. This value may need tweaking in the + # future, possibly per-test to be optimal. This works for "help" + # on board "seaboard". + self.p.timeout = 30000 + self.p.logfile_read = self.logstream + if self.config.buildconfig.get("CONFIG_SPL", False) == "y": + self.p.expect([pattern_uboot_spl_signon]) + self.uboot_spl_signon = self.p.after + self.uboot_spl_signon_escaped = re.escape(self.p.after) + else: + self.uboot_spl_signon = None + self.p.expect([pattern_uboot_main_signon]) + self.uboot_main_signon = self.p.after + self.uboot_main_signon_escaped = re.escape(self.p.after) + while True: + match = self.p.expect([self.prompt_escaped, + pattern_stop_autoboot_prompt]) + if match == 1: + self.p.send(chr(3)) # CTRL-C + continue + break + self.at_prompt = True + self.at_prompt_logevt = self.logstream.logfile.cur_evt + except Exception as ex: + self.log.error(str(ex)) + self.cleanup_spawn() + raise + + def cleanup_spawn(self): + try: + if self.p: + self.p.close() + except: + pass + self.p = None + + def validate_main_signon_in_text(self, text): + assert(self.uboot_main_signon in text) + + def disable_check(self, check_type): + return ConsoleDisableCheck(self, check_type) + + def find_ram_base(self): + if self.config.buildconfig.get("config_cmd_bdi", "n") != "y": + pytest.skip("bdinfo command not supported") + if self.ram_base == -1: + pytest.skip("Previously failed to find RAM bank start") + if self.ram_base is not None: + return self.ram_base + + with self.log.section("find_ram_base"): + response = self.run_command("bdinfo") + for l in response.split("\n"): + if "-> start" in l: + self.ram_base = int(l.split("=")[1].strip(), 16) + break + if self.ram_base is None: + self.ram_base = -1 + raise Exception("Failed to find RAM bank start in `bdinfo`") + + return self.ram_base diff --git a/test/py/uboot_console_exec_attach.py b/test/py/uboot_console_exec_attach.py new file mode 100644 index 000000000000..0267ae4dc070 --- /dev/null +++ b/test/py/uboot_console_exec_attach.py @@ -0,0 +1,36 @@ +# Copyright (c) 2015 Stephen Warren +# Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0 + +from ubspawn import Spawn +from uboot_console_base import ConsoleBase + +def cmdline(app, args): + return app + ' "' + '" "'.join(args) + '"' + +class ConsoleExecAttach(ConsoleBase): + def __init__(self, log, config): + # The max_fifo_fill value might need tweaking per-board/-SoC? + # 1 would be safe anywhere, but is very slow (a pexpect issue?). + # 16 is a common FIFO size. + # HW flow control would mean this could be infinite. + super(ConsoleExecAttach, self).__init__(log, config, max_fifo_fill=16) + + self.log.action("Flashing U-Boot") + cmd = ["uboot-test-flash", config.board_type, config.board_identity] + runner = self.log.get_runner(cmd[0]) + runner.run(cmd) + runner.close() + + def get_spawn(self): + args = [self.config.board_type, self.config.board_identity] + s = Spawn(["uboot-test-console"] + args) + + self.log.action("Resetting board") + cmd = ["uboot-test-reset"] + args + runner = self.log.get_runner(cmd[0]) + runner.run(cmd) + runner.close() + + return s diff --git a/test/py/uboot_console_sandbox.py b/test/py/uboot_console_sandbox.py new file mode 100644 index 000000000000..67fcbde11f73 --- /dev/null +++ b/test/py/uboot_console_sandbox.py @@ -0,0 +1,31 @@ +# Copyright (c) 2015 Stephen Warren +# Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0 + +import time +from ubspawn import Spawn +from uboot_console_base import ConsoleBase + +class ConsoleSandbox(ConsoleBase): + def __init__(self, log, config): + super(ConsoleSandbox, self).__init__(log, config, max_fifo_fill=1024) + + def get_spawn(self): + return Spawn([self.config.build_dir + "/u-boot"]) + + def kill(self, sig): + self.ensure_spawned() + self.log.action("kill %d" % sig) + self.p.kill(sig) + + def validate_exited(self): + p = self.p + self.p = None + for i in xrange(100): + ret = not p.isalive() + if ret: + break + time.sleep(0.1) + p.close() + return ret diff --git a/test/py/ubspawn.py b/test/py/ubspawn.py new file mode 100644 index 000000000000..3a668bc5cb65 --- /dev/null +++ b/test/py/ubspawn.py @@ -0,0 +1,97 @@ +# Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved. +# +# SPDX-License-Identifier: GPL-2.0 + +import os +import re +import pty +import select +import time + +class Timeout(Exception): + pass + +class Spawn(object): + def __init__(self, args): + self.waited = False + self.buf = "" + self.logfile_read = None + self.before = "" + self.after = "" + self.timeout = None + + (self.pid, self.fd) = pty.fork() + if self.pid == 0: + try: + os.execvp(args[0], args) + except: + print "CHILD EXECEPTION:" + import traceback + traceback.print_exc() + finally: + os._exit(255) + + self.poll = select.poll() + self.poll.register(self.fd, select.POLLIN | select.POLLPRI | select.POLLERR | select.POLLHUP | select.POLLNVAL) + + def kill(self, sig): + os.kill(self.pid, sig) + + def isalive(self): + if self.waited: + return False + + w = os.waitpid(self.pid, os.WNOHANG) + if w[0] == 0: + return True + + self.waited = True + return False + + def send(self, data): + os.write(self.fd, data) + + def expect(self, patterns): + for pi in xrange(len(patterns)): + if type(patterns[pi]) == type(""): + patterns[pi] = re.compile(patterns[pi]) + + try: + while True: + earliest_m = None + earliest_pi = None + for pi in xrange(len(patterns)): + pattern = patterns[pi] + m = pattern.search(self.buf) + if not m: + continue + if earliest_m and m.start() > earliest_m.start(): + continue + earliest_m = m + earliest_pi = pi + if earliest_m: + pos = earliest_m.start() + posafter = earliest_m.end() + 1 + self.before = self.buf[:pos] + self.after = self.buf[pos:posafter] + self.buf = self.buf[posafter:] + return earliest_pi + events = self.poll.poll(self.timeout) + if not events: + raise Timeout() + c = os.read(self.fd, 1024) + if not c: + raise EOFError() + if self.logfile_read: + self.logfile_read.write(c) + self.buf += c + finally: + if self.logfile_read: + self.logfile_read.flush() + + def close(self): + os.close(self.fd) + for i in xrange(100): + if not self.isalive(): + break + time.sleep(0.1) -- 2.6.3 _______________________________________________ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot