On 12/19/2015 03:24 PM, Simon Glass wrote:
Hi Stephen,
On 2 December 2015 at 15:18, Stephen Warren <swar...@wwwdotorg.org> wrote:
This tool aims to test U-Boot by executing U-Boot shell commands using the
console interface. A single top-level script exists to execute or attach
to the U-Boot console, run the entire script of tests against it, and
summarize the results. Advantages of this approach are:
- Testing is performed in the same way a user or script would interact
with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself.
It is asserted that writing test-related code in Python is simpler and
more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.
A few simple tests are provided as examples. Soon, we should convert as
many as possible of the other tests in test/* and test/cmd_ut.c too.
In the future, I hope to publish (out-of-tree) the hook scripts, relay
control utilities, and udev rules I will use for my own HW setup.
See README.md for more details!
This is a huge step forward for testing in U-Boot. Congratulations on
putting this together!
Tested on chromebook_link, sandbox
Tested-by: Simon Glass <s...@chromium.org>
I've made various comments in the series as I think it needs a little
tuning. I'm also interested in how we can arrange for the existing
unit tests to be run (and results supported) by this framework.
One concern I have is about the ease of running and writing tests. It
is pretty easy at present to run a particular driver model test:
./u-boot -d test.dtb -c "ut dm uclass"
and we can run this in gdb and figure out where things are going wrong
(I do this quite a bit). Somehow we need to preserve this ease of use.
The tests should be accessible. I'm not sure how you intend to make
that work.
You can select which individual tests to run by passing "-k testname" on
the command-line. Assuming Python-level tests have the desired
granularity, that should be enough for test selection.
There's currently no support for running U-Boot under gdb. The acts of
running U-Boot under gdb, disabling stdout redirection, and removing
timeouts would be pretty easy to add. However, I'm not sure how you'd be
able to interact with the gdb console and U-Boot output without
interfering with the test script's need to capture the shell output in
order to run the tests and interpret the results. Perhaps simplest would
be to add some mechanism to pause the test process so that you could
manually attach a gdb process externally at the appropriate time,
perhaps along with the test process automatically spawning another
shell/terminal/... to host the gdb (in which case it could be made to
automatically attach to the correct process).
diff --git a/test/py/.gitignore b/test/py/.gitignore
+## Testing sandbox
+
+To run the testsuite on the sandbox port (U-Boot built as a native user-space
+application), simply execute:
+
+```
+./test/py/test.py --bd sandbox --build
+```
+
+The `--bd` option tells the test suite which board type is being tested. This
+lets the test suite know which features the board has, and hence exactly what
+can be tested.
Can we use -b to fit in with buildman and patman?
pytest reserves all lower-case single-letter options for itself, so we
cannot use any of those. I suppose we could write a wrapper script to
translate from -b to --board etc., but that would be a bit messy and
prevent access to the shadowed pytest options.
diff --git a/test/py/conftest.py b/test/py/conftest.py
+def pytest_configure(config):
This series should have function comments throughout on non-trivial
functions - e.g. purpose of the function and a description of the
parameters and return value.
I can see this makes sense in some cases, but this particular function
is a standard pytest function and already documented on
http://pytest.org/. Would you still want additional documentation even
for such functions?
diff --git a/test/py/uboot_console_base.py b/test/py/uboot_console_base.py
+class ConsoleBase(object):
+ def ensure_spawned(self):
+ if self.p:
+ return
+ try:
+ self.at_prompt = False
+ self.log.action("Starting U-Boot")
+ self.p = self.get_spawn()
+ # Real targets can take a long time to scroll large amounts of
+ # text if LCD is enabled. This value may need tweaking in the
+ # future, possibly per-test to be optimal. This works for "help"
+ # on board "seaboard".
+ self.p.timeout = 30000
+ self.p.logfile_read = self.logstream
Also I have found that tests fail on chromebook_link because it cannot
keep up with the pace of keyboard input. I'm not sure what the
solution is - maybe the best thing is to implement buffering in the
serial uclass, assuming that fixes it. For now I disabled LCD output.
I think it would be worth adding a test that checks for the banner and
the prompt, so we know that other test failures are not due to this
problem.
I had the same problem on Seaboard. I solved that by having the "expect"
code wait for command echo bit-by-bit so that the host's TX side could
never overflow the target's RX side FIFOs. Perhaps try lowering the
max_fifo_fill value in test/py/uboot_console_exec_attach.py; does that
solve the issue?
_______________________________________________
U-Boot mailing list
U-Boot@lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot
