On 23 April 2014 10:26, Stoicescu, CorneliuX <corneliux.stoice...@intel.com> wrote: > Hello, > > During the 1.6 development/testing cycle we introduced and used oe-selftest > more and more for QA purposes to identify various issues in the master > branch. For those who don't know yet what is oe-selftest, it is a python unit > test based testing framework designed to simulate poky external usage > patterns. What we basically do is translate test case steps into python > classes and methods. This is useful both in reducing QA workload and the > coverage of our tests. For more information please visit the oe-selftest wiki > page: https://wiki.yoctoproject.org/wiki/Oe-selftest > > What we would like oe-selftest to be useful with as well is helping > developers quickly test their patches before submission. With this in mind, > we came up with some ideas but we also encourage everyone to contribute with > theirs. > > Test suites can be created from existing automated tests and accessed using > command-line options. For example, using 'oe-selftest --test-suite > type=recipe target=man' would test the recipe man and 'oe-selftest > --test-suite type=class target=buildhistory' would test the buildhistory > class. > > 1) Testing recipes updates or new recipes > > Even though we cannot test every single scenario or the functionality of a > recipe, we could create a test suite that would: > - build the recipe with all major architectures(qemux86, qemux86-64, qemuarm, > qemuppc, qemumips) > - rebuild the recipe from sstate(with or without a sstate file for the recipe) > - perform cleaning operations on the recipe(cleansstate) > - force all major tasks on the recipe (bitbake -C <task> <recipe>) > - selectively use each combination of .bbappend files with the recipe; all > the combinations should not break the recipe build. > - we could also create mini test suites just for some of these tests like > testing only the rebuild from sstate. > (any experience from common recipe build fails can be helpful here) >
I'd really like to see this, it could cut down on cases where something works for one developer but then hits an edge case when sent to the autobuilder. I can also see 'type=package' being useful, to check that a generated package installs cleanly in an image. For example, if package x contains a file which clashes with something already installed on core-image-minimal, 'bitbake x' won't show this but 'bitbake core-image-minimal' with 'IMAGE_INSTALL += x' set will do. Taking that a step further, that image could then be booted in qemu and the relevant ptest suite ran. I'm not sure how that would be automated myself or how the ptest log and results would be pulled off the qemu image but if it is possible I think it would be a very useful option. I also think it would be helpful to have options for a 'quick' test and a 'full' test. For some of my projects I have recipes which aren't pushed upstream to oe-core, meta-oe, etc and have a fairly restricted use-case. A quick test that just targets the current value of MACHINE and maybe skips some tests which would be expected to take a longer time could be very helpful. Thanks, -- Paul Barker Email: p...@paulbarker.me.uk http://www.paulbarker.me.uk -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
