I've created https://bugzilla.yoctoproject.org/show_bug.cgi?id=12434 to track this issue.
I get this QA error now 100% reproducible when the "right" do_package sstate archive is being used, but still haven't found the root cause (aka why/how this bad sstate archive was created in first place). On Mon, Oct 16, 2017 at 5:50 PM, Martin Jansa <martin.ja...@gmail.com> wrote: > Yes, package-output.lock is very often the cause for "^inode mismatch": > > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep '^inode mismatch' | > wc -l > 1485 > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep '^inode mismatch' | > grep package-output.lock | wc -l > 1168 > > log.do_install also causes "^inode mismatch" in many cases: > > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep '^inode mismatch' | > grep -v package-output.lock | wc -l > 317 > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep '^inode mismatch' | > grep -v package-output.lock | grep temp/log.do_install | wc -l > 309 > > That leaves only couple other files: > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep '^inode mismatch' | > grep -v package-output.lock | grep -v temp/log.do_install > inode mismatch: '/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te- > oe-linux-gnueabi/qtbase/5.9.2+gitAUTOINC+73573fce29-r0/ > image/usr/include/qt5/QtNetwork/qt_temp.rhjRGj' ino 691255 in db, 691256 > in request. > inode mismatch: '/home/jenkins/.python-history' ino 1066494 in db, > 1066493 in request. > inode mismatch: '/home/jenkins/.python-history' ino 1066494 in db, > 1066493 in request. > inode mismatch: '/home/jenkins/.python-history' ino 1066729 in db, > 1062665 in request. > inode mismatch: '/home/jenkins/.python-history' ino 1066729 in db, > 1062665 in request. > inode mismatch: '/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te- > oe-linux-gnueabi/qt3d/5.9.2+gitAUTOINC+9d8c9ada16-r0/ > image/usr/include/qt5/Qt3DInput/qt_temp.rhjRGj' ino 520536 in db, 520539 > in request. > inode mismatch: '/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te- > oe-linux-gnueabi/libqofono/0.87+gitrAUTOINC+54435de3be-r0/ > image/usr/lib/libqofono-qt5/tests/qt_temp.rhjRGj' ino 876399 in db, > 876404 in request. > inode mismatch: '/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te- > oe-linux-gnueabi/libqofono/0.87+gitrAUTOINC+54435de3be-r0/ > image/usr/lib/libqofono-qt5/tests/qt_temp.rhjRGj' ino 876404 in db, > 876408 in request. > > Similarly with "creat for", but there is more wide selection of files > triggering this one: > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep > -v 'inode mismatch' | grep 'creat for.* replaces' | wc -l > 1771 > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep > -v 'inode mismatch' | grep 'creat for' | grep package-output.lock | wc -l > 198 > > for "creat for" the log.do_install appears only twice, but various > temp/run.* files are quite common cause: > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep > -v 'inode mismatch' | grep 'creat for.* replaces' | grep temp/run | wc -l > 294 > > pm-qa is triggering this on many files, e.g. /home/jenkins/oe/world/ > shr-core/tmp-glibc/work/core2-64-oe-linux/pm-qa/0.5.2-r0/ > image/usr/bin/cpuhotplug_07.sh > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep > -v 'inode mismatch' | grep 'creat for.* replaces' | grep temp/run -v | > grep pm-qa | wc -l > 229 > > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep > -v 'inode mismatch' | grep 'creat for.* replaces' | grep temp/run -v | > grep pm-qa | grep cpuhotplug_05.sh > creat for '/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te- > oe-linux-gnueabi/pm-qa/0.5.2-r0/image/usr/bin/cpuhotplug_05.sh' replaces > existing 902737 ['NAMELESS FILE']. > creat for '/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te- > oe-linux-gnueabi/pm-qa/0.5.2-r0/image/usr/bin/cpuhotplug_05.sh' replaces > existing 1264757 ['NAMELESS FILE']. > creat for '/home/jenkins/oe/world/shr-core/tmp-glibc/work/i586-oe- > linux/pm-qa/0.5.2-r0/image/usr/bin/cpuhotplug_05.sh' replaces existing > 976636 ['NAMELESS FILE']. > creat for '/home/jenkins/oe/world/shr-core/tmp-glibc/work/core2-64- > oe-linux/pm-qa/0.5.2-r0/image/usr/bin/cpuhotplug_05.sh' replaces existing > 576004 ['NAMELESS FILE']. > creat for '/home/jenkins/oe/world/shr-core/tmp-glibc/work/core2-64- > oe-linux/pm-qa/0.5.2-r0/image/usr/bin/cpuhotplug_05.sh' replaces existing > 986226 ['NAMELESS FILE']. > > then stamps files > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep > -v 'inode mismatch' | grep 'creat for.* replaces' | grep temp/run -v | > grep -v pm-qa | grep -v package-output.lock | grep glibc/stamps | wc -l > 130 > > so it looks pretty rancom, but most of these errors are replacing > 'NAMELESS FILE' > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep > -v 'inode mismatch' | grep 'creat for.* replaces' | grep 'NAMELESS FILE' | > wc -l > 1632 > > which leaves only 139 other files: > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep > -v 'inode mismatch' | grep 'creat for.* replaces' | grep > 'replaces.*NAMELESS FILE' -v | wc -l > 139 > > from which couple of them are package-output.lock again: > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep > -v 'inode mismatch' | grep 'creat for.* replaces' | grep > 'replaces.*NAMELESS FILE' -v | grep replaces.*package-output.lock | wc -l > 31 > > and then random selection of builds, most of them from glibc-locale: > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep > -v 'inode mismatch' | grep 'creat for.* replaces' | grep > 'replaces.*NAMELESS FILE' -v | grep replaces.*package-output.lock -v | grep > glibc-locale | wc -l > 34 > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep > -v 'inode mismatch' | grep 'creat for.* replaces' | grep > 'replaces.*NAMELESS FILE' -v | grep replaces.*package-output.lock -v | grep > llvm | wc -l > 3 > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep > -v 'inode mismatch' | grep 'creat for.* replaces' | grep > 'replaces.*NAMELESS FILE' -v | grep replaces.*package-output.lock -v | grep > linux-yocto | wc -l > 7 > > "^creat ignored" is again mostly 195/202 package-output.lock: > > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep > -v 'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates | > grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep > -v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep > '^creat ignored ' | wc -l > 202 > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep > -v 'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates | > grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep > -v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep > '^creat ignored ' | grep package-output.lock | wc -l > 195 > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep > -v 'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates | > grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep > -v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep > '^creat ignored ' | grep package-output.lock -v > creat ignored for existing file '/home/jenkins/oe/world/shr- > core/tmp-glibc/work/armv5te-oe-linux-gnueabi/qtbase/5.9.2+ > gitAUTOINC+73573fce29-r0/image/usr/include/qt5/QtNetwork/qt_temp.rhjRGj'. > creat ignored for existing file '/home/jenkins/oe/world/shr- > core/tmp-glibc/work/armv5te-oe-linux-gnueabi/qt3d/5.9.2+ > gitAUTOINC+9d8c9ada16-r0/image/usr/include/qt5/Qt3DInput/qt_temp.rhjRGj'. > creat ignored for existing file '/home/jenkins/oe/world/shr- > core/tmp-glibc/work/armv5te-oe-linux-gnueabi/libqofono/0. > 87+gitrAUTOINC+54435de3be-r0/image/usr/lib/libqofono-qt5/ > tests/qt_temp.rhjRGj'. > creat ignored for existing file '/home/jenkins/oe/world/shr- > core/tmp-glibc/work/armv5te-oe-linux-gnueabi/libqofono/0. > 87+gitrAUTOINC+54435de3be-r0/image/usr/lib/libqofono-qt5/ > tests/qt_temp.rhjRGj'. > creat ignored for existing file '/home/jenkins/oe/world/shr- > core/tmp-glibc/sstate-control/index-armv5te.lock'. > creat ignored for existing file '/home/jenkins/oe/world/shr- > core/tmp-glibc/sstate-control/index-armv5te.lock'. > creat ignored for existing file '/home/jenkins/oe/world/shr- > core/tmp-glibc/work/i586-oe-linux/usermode/1.111-r0/build/ > po/.intltool-merge-cache'. > > ^dir err: also happens a lot in the log files where other issues appear as > well (because the QA check isn't triggered by this one): > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep > -v 'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates | > grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep > -v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep > '^creat ignored ' -v | grep '^dir err ' | wc -l > 744 > > couple ^symlink err: > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep > -v 'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates | > grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep > -v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep > '^creat ignored ' -v | grep '^dir err ' -v | grep '^symlink err' | wc -l > 223 > > martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep > -v 'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates | > grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep > -v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep > '^creat ignored ' -v | grep '^dir err ' -v | grep '^symlink err' -v | grep > '^symlink mismatch: ' | wc -l > 309 > > I fail to see useful patterns in this, maybe fixing the > package-output.lock might help to filter the noise a bit. > > On Mon, Oct 16, 2017 at 5:10 PM, Burton, Ross <ross.bur...@intel.com> > wrote: > >> On 16 October 2017 at 16:01, Martin Jansa <martin.ja...@gmail.com> wrote: >> >>> The world builds with this check applied show that *a lot* of components >>> report one of these errors and often many of errors, the most complete logs >>> from last couple days: >>> >> >> A lot of the errors in my logs were due to package-output.lock being >> accessed inside/outside pseudo context. Can you grep all of your logs and >> collate some common patterns? >> >> Ross >> > >
-- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
