Hi all, I started this outline of my "smoketest" plans as a reply to David, but it grew enough to deserve spawning its own new thread.
On Sun, 2009-05-24 at 14:20 -0700, David Brownell wrote: > On Saturday 23 May 2009, Zach Welch wrote: > > 5) commit testing tools > > - one-step smoke tests! I probably need another week for this. > > - all in-tree with no new dependencies (maybe a Perl module or two) > > - > > http://www.mail-archive.com/openocd-development@lists.berlios.de/msg04030.html > > What's the future of the existing testing/* stuff by the way? > Not that I've done more than glance at it ... I am hoping that parts of it can be integrated at the lowest level, but it would take a lot of work to get it to where I want. I haven't figured out how to run those tests (much less extend them), so I think it will be difficult for users to try them out as it stands. I view each user's contribution as only part of a single smoke-test. In other words, accurate test results will only be derivable after amassing multitudes of reports from many different users. Further, I want to allow users collect to their own reports and data, as it should be possible to extend the smoke-test for custom work. This seeks to provide the same extensibility for testing as the configuration files that we want to give interface/chip/target/board support. For the data set to be useful, these results need to be easy and fast to produce (and include all of the information required to diagnose frequently seen problems), and the results need to be available for others to review instantly. Users making (or collecting) these reports should need only their SVN working copy and standard development tools, but the data should not be required to be stored in the repository. In parallel and conjunction with my Support DB work, here are some things that I have vaguely planed to implement in this area: - extend 'make check' with a smoketest app - checks for OOCD_TEST_CONFIG, etc. in environment (or config file) - if properly set, runs the smoke test with specified parameters - openocd -f ${OOCD_TEST_CONFIG} - implies a modular test suite (see below) - should be able to run some minimal tests with dummy interface: - compare results of baseline sanity checks with expected results - builds a more complete test suite: - existing testing/examples/ look like a great start - all targets should be tested fully and for all capabilities - we do NOT want a "lowest common denominator" test suite - ... but can we start with one to get going? - probably requires one test configuration file per board/target - modularization can occur here, just like with targets/boards/chips - coverage can increase over time, building up bundles of tests - add 'smoketest' Makefile target: - calls 'make check' (and the smoketest app) - gather inputs and output into a report file - add targets to post the report: - 'smokereport' -- send to list via e-mail (via sendmail) - 'smokepost' -- send web form (via curl or other script) - automatically build tool-chains required for cross-compiling - produce mingw32, arm-elf, others using in-tree scripts - build all required target code from sources If I get a little bit further, I will be able to address all of these issues in much greater detail. I am hoping the original author of the 'testing/' code will step forward and contribute their feedback; I have started to eyeball its documentation for incorporation into The Manual. If nothing else, Øyvind added the new selftest.cfg to the tree, so I expect that he will want to contribute to this effort when he gets back from his vacation. I would like to try to get some or all of these features in place for 0.2.0, since these should not affect the core functionality of OpenOCD in any respect. Cheers, Zach _______________________________________________ Openocd-development mailing list Openocd-development@lists.berlios.de https://lists.berlios.de/mailman/listinfo/openocd-development
