Hi Here is the report for Full QA Cycle on Release 2.3 M3 rc1
Full Report : https://wiki.yoctoproject.org/wiki/WW11_-_2017-03-15_-_Full_Test_Cycle_2.3_M3_rc1 Summary The QA cycle for point release 2.3 M3 rc1 .1 is complete. There are 12 new issues found, 4 of them are High and 5 are M+. 2 of the high were blocking Performance and Selftest execution and were quickly fixed and merged, hence those 2 test suites were executed in a different commit from the one tagged for M3 rc1. In total there are 20 bugs attached to 2.3 M3 rc1, 5 High, 7 M+ and 8 M. Also there are still 15 M+ to be implemented still tagged to 2.3 M3 release. Regarding to Performance there is huge improvements on rootfs measurements on both machines Fedora and Ubuntu and eSDK on Ubuntu, and some little regressions (around 3%) on both machines in different tasks. pTest shows improvements on acl and e2fsprogs moudles but also shows regression on other modules. QA Hints - There are still may high and M+ bugs that are opened against M3 rc1 - 2 of the highest test suites (Selftest and Performance) were not executed against M3 rc1 commit, even when they did show up good results on the commit were tested on the commit tagged for 2.3 M3 rc1 are failing. - There are still 15 M+ tagged to be implemented on 2.3 M3, hence a decision should be taken about them, implemented for M4 or moved to 2.4 Based on above points the strategy to follow up form QA perspective is to have another release candidate when the blocker issues get solved and hence have a better quality on M3 and avoid High issues on M4. === Details === Bugs * New - High o 11154 - buildperf: Fedora machine failing on buildperf execution due could not find 'ip' tool [1] o 11172 - bitbake: Fetcher failure when building core-image-minimal with "ERROR: socat: not found" [2] o 11193 - runqemu cannot launch image if enable "rm_work" [3] o 11223 - devtool runqemu doesn't work in eSDK [4] - M+ o 11224 - Compliance: LTP, POSIX, LSB tests are not creating results file (.targz) [5] o 11222 - [eclipse] C compiler cannot create executables when reconfiguring project [6] o 11174 - Update yocto-bsp script for kernel 4.10 [7] o 11144 - Build Appliance : Toaster requires pod2man [8] o 11185 - selftest/devtool: failing when parsing "mraa" recipe [9] - M / Low o 11173 testimage: "test_dnf_install_from_http" is failing [10] o 11220 runqemu: Qemux86 not boot with Ubuntu [11] o 11225 Build from Command line fail when finish all the tasks [12] * High / M+ Not New - 11119 runqemu-extract-sdk is failing when extracting .tar.bz file [13] - 11064 [Yocto 2.2.1] kernel version mismatched when using yocto-bsp to create custom qemu bsp [14] - 11025 selftest/devtool: Build mdadm failing on Centos7 and Ubuntu 1404 [15] Full bugs Report : https://wiki.yoctoproject.org/wiki/WW11_-_2017-03-15_-_Full_Test_Cycle_2.3_M3_rc1#Bugs_Found_during_QA_Test Performance Performance measurements were huge improved for rootfs on both machines (Ubuntu around 30%, Fedora 20%) falling down the numbers to the average observed before the "recipe specific sysroot" implementation. On Ubuntu there was also a very good improvement on eSDK deploy time of around 12%, some other measurement times were up on around 3% on both machines, below are the tables with the times of the tests: - Ubuntu Test 2.3 M2 rc3 2.3 M3 rc1 % sato 1:11:47 1:10:15 -2.14 rootfs 4:02 2:28 -38.84 rmwork 1:05:56 1:05:28 -0.71 kernel 5:00 5:11 +3.67 eSDK 3:28 3:01 -12.98 - Fedora Test 2.3 M2 rc3 2.3 M3 rc1 % sato 1:12:58 1:12:34 -0.55 rootfs 3:19 2:40 -19.60 rmwork 1:04:50 1:07:10 +3.60 kernel 6:27 6:29 +0.52 eSDK 3:11 3:17 +3.14 Performance Charts : https://wiki.yoctoproject.org/charts/perf_milestone_GDC/performance_test.html pTest - There were improvemnte of pass rate for acl and e2fsprogs modules - Modules that decreased pass rate were glib-2.0, openssh, openssl, quilt, strace, valgrind - A new bug was raised to address quil failures [15] pTest full report : https://wiki.yoctoproject.org/wiki/WW11_-_2017-03-15_-_Full_Test_Cycle_2.3_M3_rc1#pTest_for_genericx86-64_on_NUC Kernel - to test case "TCTEMP_2.3_MANUAL_Kernel_dev_External_source" was run, however due an issue with git, currently there is no way to test with 100% of reliability. I also tested on different machines. An empty directory is released when you are trying to clone from remote and local. I tested it also with older kernel 4.4, 4.6 and I got the same issue. One month ago I could reproduce this properly. Links 1. https://bugzilla.yoctoproject.org/show_bug.cgi?id=11154 2. https://bugzilla.yoctoproject.org/show_bug.cgi?id=11172 3. https://bugzilla.yoctoproject.org/show_bug.cgi?id=11193 4. https://bugzilla.yoctoproject.org/show_bug.cgi?id=11223 5. https://bugzilla.yoctoproject.org/show_bug.cgi?id=11224 6. https://bugzilla.yoctoproject.org/show_bug.cgi?id=11222 7. https://bugzilla.yoctoproject.org/show_bug.cgi?id=11174 8. https://bugzilla.yoctoproject.org/show_bug.cgi?id=11144 9. https://bugzilla.yoctoproject.org/show_bug.cgi?id=11185 10. https://bugzilla.yoctoproject.org/show_bug.cgi?id=11173 11. https://bugzilla.yoctoproject.org/show_bug.cgi?id=11220 12. https://bugzilla.yoctoproject.org/show_bug.cgi?id=11225 13. https://bugzilla.yoctoproject.org/show_bug.cgi?id=11119 14. https://bugzilla.yoctoproject.org/show_bug.cgi?id=11064 15. https://bugzilla.yoctoproject.org/show_bug.cgi?id=11025 16. https://bugzilla.yoctoproject.org/show_bug.cgi?id=11221 Regards, José
-- _______________________________________________ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
