Hey guys, On Sat, Jul 6, 2019 at 5:52 AM Richard Purdie < richard.pur...@linuxfoundation.org> wrote:
> On Sat, 2019-07-06 at 11:39 +0000, Nathan Rossi wrote: > > This patch is an RFC for adding support to execute the gnu test suites > for > > binutils, gcc and glibc. With the intention for enabling automated test > running > > of these test suites within the OEQA framework such that they can be > executed by > > the Yocto Autobuilder. > > > > Please note that this patch is not a complete implementation and needs > > additional work as well as changes based on comments and feedback from > this RFC. > > This is rather cool, thanks! > > Looking at this was on my todo list once we got the existing OEQA, > ptest and ltp setups working well. I'm very happy to have been beaten > to it though. > > > The test suites covered need significant resources or build artifacts > such > > that running them on the target is undesirable which rules out the use > of ptest. > > Because of this the test suites can be run on the build host and if > necessary > > call out to the target. > > > > The following implementation creates a number of recipes that are used to > > build/execute the test suites for the different components. The reason > for > > creating separate recipes is primarily due to dependencies and the need > for > > components in the sysroot. For example binutils has tests that use the C > > compiler however binutils is a dependency for the C compiler and thus > would > > cause a dependency loop. The issue with sysroots occurs with dependence > on > > `*-initial` recipes and the test suites needing the non-initial version. > > I think this means you're working with something pre-warrior as we got > rid of most of the *-initial recipes apart from libgcc-initial. > Yup, I agree with this, and yes, we still have initial recipes, which is in what Nathan based his work. > > Some issues with splitting the recipes: > > - Rebuilds the recipe > > - Like gcc-cross-testsuite in this patch, could use a stashed builddir > > - Source is duplicated > > - gcc gets around this with shared source > > - Requires having the recipe files and maintaining them > > - Multiple versions of recipes > > - Multiple variants of recipes (-cross, -crosssdk, -native if desired) > > It might be possible to have multiple tasks in these recipes and have > the later tasks depend on other pieces of the system like the C > compiler, thereby avoiding the need for splitting if only the later > tasks have the dependencies. Not sure if it would work or not but may > be worth exploring. > Worth exploring but might end up being more convoluted than necessary IMO. Benefit vs Complication issue. > > Target execution is another issue with the test suites. Note that > binutils > > however does not require any target execution. In this patch both > > qemu-linux-user and ssh target execution solutions are provided. For the > > purposes of OE, qemu-linux-user may suffice as it has great success at > executing > > gcc and gcc-runtime tests with acceptable success at executing the glibc > tests. > > I feel fairly strongly that we probably want to execute these kinds of > tests under qemu system mode, not the user mode. The reason is that we > want to be as close to the target environment as we can be and that > qemu-user testing is at least as much of a test of qemu's emulation > that it is the behaviour of the compiler or libc (libc in particular). > I was thinking this and then later read you confirmed my suspicions > below... > I believe the QEMU recipe splitting is also new in the tree, and Nathan isn't basing his work on that, so there might be some issues there. > > > The glibc test suite can be problematic to execute for a few reasons: > > - Requires access to the exact same filesystem as the build host > > - On physical targets and QEMU this requires NFS mounts > > We do have unfs support already under qemu which might make this > possible. > > > - Relies on exact syscall behaviour > > - Causes some issues where there are differences between > qemu-linux-user and > > the target architectures kernel > > Right, this one worries me and pushes me to want to use qemu system > mode. > > > - Can consume significant resources (e.g. OOM, or worse trigger > bugs/panics in > > kernel drivers) > > Any rough guide to what significant is here? ptest needs 1GB memory for > example. qemu-system mode should limit that to the VMs at least? > > > - Slow to execute > > - With QEMU system emulation it can take many hours > > We do have KVM acceleration for x86 and arm FWIW which is probably > where we'd start testing this on the autobuilder. > Excuse me if I'm mistaken, but would this be something similar to what we did for python3 optimization? > > > - With some physical target architectures it can take days (e.g. > microblaze) > > > > The significantly increased execution speed of qemu-linux-user vs qemu > system > > with glibc, and the ability for qemu-linux-user to be executed in > parallel with > > the gcc test suite makes it a strong solution for continuous integration > > testing. > > Was that with or without KVM? > > > The following table shows results for the major test suite components > running > > with qemu-linux-user execution. The numbers represent 'failed > tests'/'total > > tests'. The machines used to run the tests are the `qemu*` machine for > the > > associated architecture, not all qemu machines available in oe-core were > tested. > > It is important to note that these results are only indicative of > > qemu-linux-user behaviour and that there are a number of test failures > that are > > due to issues not specific to qemu-linux-user. > > > > | gcc | g++ | libstdc++ | binutils | gas > | ld | glibc > > x86-64 | 589/135169 | 457/131913 | 1/13008 | 0/ 236 | > 0/ 1256 | 166/ 1975 | 1423/ 5991 > > arm | 469/123905 | 365/128416 | 19/12788 | 0/ 191 | > 0/ 872 | 155/ 1479 | 64/ 5130 > > aarch64 | 460/130904 | 364/128977 | 1/12789 | 0/ 190 | > 0/ 442 | 157/ 1474 | 76/ 5882 > > powerpc | 18336/116624 | 6747/128636 | 33/12996 | 0/ 187 | > 1/ 265 | 157/ 1352 | 1218/ 5110 > > mips64 | 1174/134744 | 401/130195 | 22/12780 | 0/ 213 | > 43/ 7245 | 803/ 1634 | 2032/ 5847 > > riscv64 | 456/106399 | 376/128427 | 1/12748 | 0/ 185 | > 0/ 257 | 152/ 1062 | 88/ 5847 > > I'd be interested to know how these numbers compare to the ssh > execution... > > The binutils results look good! :) > > This is awesome!, some are a little scary though (percentage wise) > > This patch also introduces some OEQA test cases which cover running the > test > > suites. However in this specific patch it does not include any > implementation > > for the automated setup of qemu system emulation testing with runqemu > and NFS > > mounting for glibc tests. Also not included in these test cases is any > known > > test failure filtering. > > The known test failure filtering is something we can use the OEQA > backend for, I'd envisage this being intergrated in a similar way to > the way we added ptest/ltp/ltp-posix there. > > > I would also be interested in the opinion with regards to whether these > test > > suites should be executed as part of the existing Yocto Autobuilder > instance. > > Short answer is yes. We won't run them all the time but when it makes > sense and I'd happily see the autobuilder apart to be able to trigger > these appropriately. We can probably run the KVM accelerated arches > more often than the others. > Would we separate test cases into different sets/suites based on importance?, and yes I'd love to see this in the Yocto AB. > Plenty of implementation details to further discuss but this is great > to see! > > Cheers, > > Richard > > This looks good, great work Nathan!, my only other comment would be that we would probably need two versions of the patches (one for thud) and one for master/warrior where some of the changes to *-initial recipes and qemu- system/user have happened already. Regards, Alejandro > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core > -- *M.S. Alejandro Enedino Herna**ndez Samaniego*
-- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
