Hi Nathan, > On Thu, 26 Aug 2021 at 23:38, Richard Purdie > <richard.pur...@linuxfoundation.org> wrote: > > > > On Thu, 2021-08-26 at 14:27 +0200, Lukasz Majewski wrote: > > > Hi Richard, Khem, > > > > > > > Hi Richard, > > > > > > > > > On Tue, 2021-08-24 at 09:36 +0200, ?ukasz Majewski wrote: > > > > > > Hi Khem, > > > > > > > > > > > > > On Mon, Aug 23, 2021 at 1:09 PM Lukasz Majewski > > > > > > > <lu...@denx.de> wrote: > > > > > > > > > > > > > > > > On Mon, 23 Aug 2021 12:52:44 -0700 > > > > > > > > Khem Raj <raj.k...@gmail.com> wrote: > > > > > > > > > > > > > > > > > On Mon, Aug 23, 2021 at 11:24 AM Lukasz Majewski > > > > > > > > > <lu...@denx.de> wrote: > > > > > > > > > > > > > > > > > > > Hi Khem, > > > > > > > > > > > > > > > > > > > > > On 8/23/21 8:08 AM, ?ukasz Majewski wrote: > > > > > > > > > > > > This patch introduces new recipe - namely > > > > > > > > > > > > 'glibc-tests', which builds and installs glibc > > > > > > > > > > > > test suite to OE/Yocto built image. > > > > > > > > > > > > > > > > > > > > > > > > It reuses code from already available > > > > > > > > > > > > 'glibc-testsuite' recipe, which is run with > > > > > > > > > > > > 'bitbake glibc-testsuite -c check' and uses > > > > > > > > > > > > qemu to execute remotely (via SSH) tests on > > > > > > > > > > > > some emulated machine. > > > > > > > > > > > > > > > > > > > > > > > > This recipe installs eligible tests on some > > > > > > > > > > > > rootfs image. Afterwards, either all of them or > > > > > > > > > > > > only time related subset, those tests can be > > > > > > > > > > > > executed on the real hardware, to facilitate > > > > > > > > > > > > validation of this platform with for example > > > > > > > > > > > > Y2038 problem compliance. > > > > > > > > > > > > > > > > > > > > > > > > By default all tests are executed, with > > > > > > > > > > > > 'ptest-runner glibc-tests'. To test only time > > > > > > > > > > > > related subset - one needs to call: cd > > > > > > > > > > > > /usr/lib/glibc-tests/ptest/; rm run-ptest; \ ln > > > > > > > > > > > > -s run-ptest-time run-ptest; cd -; ptest-runner > > > > > > > > > > > > glibc-tests > > > > > > > > > > > > > > > > > > > > > > > > To facilitate debugging, source files are > > > > > > > > > > > > provided by default with the unstripped > > > > > > > > > > > > debugging symbols. Such approach would reduce > > > > > > > > > > > > the already complex recipe (as it inherits base > > > > > > > > > > > > glibc one), so there is no need to also install > > > > > > > > > > > > *-dbg and *-src packages. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > does it have to be a separate recipe I wonder if > > > > > > > > > > > we can have it built as part of glibc itself > > > > > > > > > > > controlled via ptest knob > > > > > > > > > > > > > > > > > > > > I've followed the glibc-testsuite recipe to provide > > > > > > > > > > tests for ptest. Test creation is similar to it, > > > > > > > > > > but doesn't require QEMU run (tests are executed on > > > > > > > > > > the HW). > > > > > > > > > > > > > > > > > > > > Another rationale was to have some kind of > > > > > > > > > > "features" separation in different file (liked > > > > > > > > > > glibc-mtrace), which is easier to maintain and > > > > > > > > > > review. > > > > > > > > > > > > > > > > > > > > Last but not least - separate recipe (and built > > > > > > > > > > binaries) allow some kind of magic with selection > > > > > > > > > > of used test programs (this may be useful if one > > > > > > > > > > would like to have different sets of tests in > > > > > > > > > > different packages) > > > > > > > > > > > > > > > > > > That’s seems ok I think the names are quite confusing > > > > > > > > > now not that it was not so much better before but now > > > > > > > > > we have glibc-tests and glibc-testsuites which do > > > > > > > > > same things but in very different way maybe > > > > > > > > > glibc-testsuite should be renamed to something like > > > > > > > > > glibc-tests-crosd or some such > > > > > > > > > > > > > > > > I think that glibc-testsuite_2.34.bb shall be renamed to > > > > > > > > glibc-tests-qemu_2.34.bb as it is more descriptive. > > > > > > > > > > > > > > is it only runnable in qemu ? I thought we could use a > > > > > > > read board as well with IP address > > > > > > > > > > > > It looks like by default the QEMU is used in this recipe. > > > > > > > > > > > > This recipe provides custom check-test-wrapper, which can > > > > > > log into remote board via ssh (when TOOLCHAIN_TEST_TARGET = > > > > > > "ssh"). > > > > > > > > > > > > This looks a bit awkward for me, as the goal would be to > > > > > > run built tests on target (exact) HW without the need of > > > > > > SSH. > > > > > > > > > > > > It is also easier to debug the code with the real HW. > > > > > > > > > > I'm a bit worried we're going around in circles with this. We > > > > > did have a different form of tests a while back but I was > > > > > told those didn't work well and they were replaced with the > > > > > current setup which worked with the driver in the glibc > > > > > source code. > > > > > > > > I've looked into the glibc-testing.inc from SHA1: > > > > 0a42ae7f7ca8fcebe4630b701e50e2352f9b7c3b > > > > > > > > There glibc tests were built and copied on the target via NFS. > > > > > > > > > This was done around > > > > > the time we added support for the other toolchain test suites > > > > > (binutils+gcc). > > > > > > > > I'm not familiar with the "binutils+gcc" issue, which was > > > > solved in the past. Could you share any reference? > > > > > > > > > We ended up with options to use full system qemu or > > > > > user mode qemu emulation depending on the speed of the target > > > > > (e.g. with kvm or not). > > > > > > > > From what I see in the recipe - you either run user mode QEMU by > > > > default (at least when I do run it): > > > > > > > > NOTE: make PARALLELMFLAGS=-j 8 SHELL=/bin/bash -i > > > > QEMU_SYSROOT=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/recipe-sysroot > > > > QEMU_OPTIONS=qemu-arm -r 5.1.0 SSH_HOST=localhost SSH_HOS > > > > T_USER=root SSH_HOST_PORT=2222 > > > > test-wrapper=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/check-test-wrapper > > > > user check > > > > > > > > Or you log to the board via SSH (the 'ssh' mode). > > > > > > > > > > > > > > Has this patchset been run with all the tests in glibc or > > > > > just a subset? > > > > > > > > This patch set allows running all tests [*], or (preferably in > > > > my use case) only a time related ones. The latter is to > > > > validate Y2038 code, which testing has two issues: > > > > > > > > 1. It cannot be tested with QEMU user mode [1]. Instead "linux > > > > timespaces" were suggested, but it lacks support for > > > > CLOCK_REALTIME manipulation for those tests [2]. > > So just a note, the qemu user mode testing is not a complete solution. > There are a number of problems with it, but in general it is still > quite useful for validating general functionally, especially when > implementing a new platform.
+1 > A big one is due to the ability to run > tests in parallel with user mode, it easily reduces test run times by > an order of magnitude (compared to qemu system, which for some > platforms is faster than real hardware). +1 > > The reason why qemu user mode is the default behaviour is because > running "bitbake glibc-testsuite -c check" will work without any > external dependencies (the remote machine, NFS/sshfs, etc.). +1 > > > > > > > > > 2. Testing with SSH is possible, but is not as robust as > > > > expected (i.e. imposes SSHD on the target board, and ETH > > > > connection). > > > > > > > > > > I'm always very nervous about having two ways to do similar > > > > > things. > > > > > > > > Technically, I just reuse the glibc-testsuite_2.34.bb to the > > > > point where tests are _only_ built. Then I pack those binaries > > > > and install on the target's rootfs to be run with ptest > > > > framework. > > So it appears the make variable "run-built-tests" excludes the > building of some tests. As fair as I remember, the 'run-built-tests=no' switch tells the glibc build system to not execute built tests. > So I am not sure if the complete test suite > can be built in this way. I'm pretty sure that all tests required for 'check' are build. > Though I can see some value in having the > glibc-testsuite recipe being able to only build the tests for > development purposes (e.g. TOOLCHAIN_TEST_TARGET = "build"?). > > When I had implemented the testsuite recipe the glibc testsuite relied > quite heavily on the environment it was built and run in, in > particular around configuration. That is probably still an issue. Indeed, some (a few?) tests rely on the build environment in which they were build. I do personally, regard this as a some kind of a bug. I do believe, that when you build test for a particular version of glibc (2.34 in this case), one shall expect that it can be run only by linking the proper library. This is the case for "time" related tests. On my test system with this patch applied, all of them are executed with no errors. Those are self contained. > > It would be interesting to know how much of the test suite can be > run/passes and how that compares to the results of existing > system/user emulation with cross execution using the glibc makefile. I can provide exact numbers when I finish porting more Y2038 related patches to newest glibc -master. > > > > > > > > > Such approach has several advantages: > > > > > > > > - No need for SSH (which can hang, as you need sshfs in user > > > > space for the setup) > > I had not tested with sshfs that likely has quite low performance, the > OEQA tests rely on NFS for QEMU system emulation and that is what I > had been using for hardware testing. I had some issues with NFS - either I couldn't place some OE/Yocto build directory in it or glibc's build directory was not working with it correctly. That was why I had to use the sshfs instead, which is _really_ slow and often breaks. > However it would be interesting > to see if virtiofs could improve the performance for testing with QEMU > system emulation. > > > > > > > > > - Run on the real HW (and environment) with the same version of > > > > glibc on the real target. > > > > > > > > - Sources and debugging symbols available on the target, so > > > > easier GDB debugging. > > > > > > > > - Using ptest framework to execute tests > > > > > > > > - Time related tests are "self contained", so could be executed > > > > without built data. > > > > > > > > At least for Y2038 validation such approach seems more > > > > appealing. > > > > > > Do you have any more feedback regarding this patch? > > > > We're at feature freeze and this wasn't a planned change so it is > > too late for this release cycle, I'm struggling with the things > > that were planned. > > > > I need to spend some time looking at what the other code is doing > > and how we're using it on the autobuilder and whether we want to > > change to use the ptest approach or not. > > > > I added Nathan Rossi to cc since he worked on the other code for > > this and may have more insight. > > Thanks for cc'ing, I somehow managed to miss the thread. > > Regards, > Nathan Best regards, Lukasz Majewski -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lu...@denx.de
pgpiF6UnPmR3a.pgp
Description: OpenPGP digital signature
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#155400): https://lists.openembedded.org/g/openembedded-core/message/155400 Mute This Topic: https://lists.openembedded.org/mt/85087396/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
