On Tue, 2024-09-17 at 07:22 -0700, philip.dawson via lists.yoctoproject.org wrote: > I'm trying set up automated CI builds of our yocto builds and am running into > intermittent issues fetching sstate cache files from our server. > We're using Xilinx Petalinux tools 2023.1 version which corresponds to Yocto > Langdale. I'm posting here rather than in meta-xilinx as this feel like a > more general issue than a Xilinx specific one. > > The build is run inside a docker image on an Ubuntu 22.04 host machine. > We have a NGINX server hosting our sstate cache and as a download pre-mirror. > The setup has been shown to work in local builds but when running from our > build server CI pipelines is unreliable. > > Even though the build completes by fetching from upstream sources it reports > as a failure and the CI pipeline aborts: > 2024-09-12T08:39:38.0284903Z NOTE: Tasks Summary: Attempted 8095 tasks of > which 6863 didn't need to be rerun and all succeeded. > 2024-09-12T08:39:38.0387164Z > 2024-09-12T08:39:38.0387977Z Summary: There were 28 WARNING messages. > 2024-09-12T08:39:38.0389108Z Summary: There were 6 ERROR messages, returning > a non-zero exit code. > 2024-09-12T08:39:38.4545026Z ERROR: Failed to build project. Check the > /__w/1/s/DS10G/yocto/dgc220_release/dgc220/build/build.log file for more > details... > > The specifics of the issue are that in any given build a handful of sstate > fetches from our server fail with errors like: > 2024-09-12T08:10:54.9631514Z NOTE: recipe libgcrypt-native-1.10.1-r0: task > do_populate_sysroot_setscene: Started > ... > 2024-09-12T08:11:19.2785546Z WARNING: libgcrypt-native-1.10.1-r0 > do_populate_sysroot_setscene: Failed to fetch URL > file://universal/a6/18/sstate:libgcrypt-native:x86_64-linux:1.10.1:r0:x86_64:10:a61860a7702fbf31e63bdcbadf5348983eecb402fb01a4de25a91d8b9e45a9a3_populate_sysroot.tar.zst.siginfo;downloadfilename=universal/a6/18/sstate:libgcrypt-native:x86_64-linux:1.10.1:r0:x86_64:10:a61860a7702fbf31e63bdcbadf5348983eecb402fb01a4de25a91d8b9e45a9a3_populate_sysroot.tar.zst.siginfo, > attempting MIRRORS if available > 2024-09-12T08:11:19.2792229Z ERROR: libgcrypt-native-1.10.1-r0 > do_populate_sysroot_setscene: Fetcher failure: Unable to find file > file://universal/a6/18/sstate:libgcrypt-native:x86_64-linux:1.10.1:r0:x86_64:10:a61860a7702fbf31e63bdcbadf5348983eecb402fb01a4de25a91d8b9e45a9a3_populate_sysroot.tar.zst.siginfo;downloadfilename=universal/a6/18/sstate:libgcrypt-native:x86_64-linux:1.10.1:r0:x86_64:10:a61860a7702fbf31e63bdcbadf5348983eecb402fb01a4de25a91d8b9e45a9a3_populate_sysroot.tar.zst.siginfo > anywhere. The paths that were searched were: > 2024-09-12T08:11:19.2793587Z > /__w/1/s/DS10G/yocto/dgc220_release/dgc220/build/sstate-cache > > In the build above 5 other packages failed to fetch at the same time (within > 50ms). The exact packages that fail and number of packages that fail vary > from build to build. > > I have managed to capture a tcpdump on the docker host machine and the access > logs of the NGINX server at the time of the above build. The access logs show > a HEAD and GET pair to check the tar.zst exists and fetch it, then HEAD > request for the tar.zst.siginfo which returns OK but no corresponding GET. > About 16s later the build errors happen. > > I can see no obvious errors in the tcpdump and no sign of failed attempts to > GET the siginfo. There are no errors reported from NGINX. > > From reading other posts I accept this is probably some issue with our > network that needs to be sorted, but I don't have much to give our IT > department to go on other than it's not working reliably. > Is there a way I could get yocto/bitbake to produce more logs or diagnostics > as to why/how the fetch of the siginfo part of the sstate cache is failing > (before the warning/error that the local file doesn't exist)? > > Or do you have any other advice that might help?
It could be: https://git.yoctoproject.org/poky/commit/bitbake/lib?id=4172c3bdd5a076da6bb125071a8157a0fef28932 Cheers, Richard
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#63855): https://lists.yoctoproject.org/g/yocto/message/63855 Mute This Topic: https://lists.yoctoproject.org/mt/108501743/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
