Hi, I'm trying set up automated CI builds of our yocto builds and am running into intermittent issues fetching sstate cache files from our server. We're using Xilinx Petalinux tools 2023.1 version which corresponds to Yocto Langdale. I'm posting here rather than in meta-xilinx as this feel like a more general issue than a Xilinx specific one.
The build is run inside a docker image on an Ubuntu 22.04 host machine. We have a NGINX server hosting our sstate cache and as a download pre-mirror. The setup has been shown to work in local builds but when running from our build server CI pipelines is unreliable. Even though the build completes by fetching from upstream sources it reports as a failure and the CI pipeline aborts: 2024-09-12T08:39:38.0284903Z NOTE: Tasks Summary: Attempted 8095 tasks of which 6863 didn't need to be rerun and all succeeded. 2024-09-12T08:39:38.0387164Z 2024-09-12T08:39:38.0387977Z Summary: There were 28 WARNING messages. 2024-09-12T08:39:38.0389108Z Summary: There were 6 ERROR messages, returning a non-zero exit code. 2024-09-12T08:39:38.4545026Z ERROR: Failed to build project. Check the /__w/1/s/DS10G/yocto/dgc220_release/dgc220/build/build.log file for more details... The specifics of the issue are that in any given build a handful of sstate fetches from our server fail with errors like: 2024-09-12T08:10:54.9631514Z NOTE: recipe libgcrypt-native-1.10.1-r0: task do_populate_sysroot_setscene: Started ... 2024-09-12T08:11:19.2785546Z WARNING: libgcrypt-native-1.10.1-r0 do_populate_sysroot_setscene: Failed to fetch URL file://universal/a6/18/sstate:libgcrypt-native:x86_64-linux:1.10.1:r0:x86_64:10:a61860a7702fbf31e63bdcbadf5348983eecb402fb01a4de25a91d8b9e45a9a3_populate_sysroot.tar.zst.siginfo;downloadfilename=universal/a6/18/sstate:libgcrypt-native:x86_64-linux:1.10.1:r0:x86_64:10:a61860a7702fbf31e63bdcbadf5348983eecb402fb01a4de25a91d8b9e45a9a3_populate_sysroot.tar.zst.siginfo, attempting MIRRORS if available 2024-09-12T08:11:19.2792229Z ERROR: libgcrypt-native-1.10.1-r0 do_populate_sysroot_setscene: Fetcher failure: Unable to find file file://universal/a6/18/sstate:libgcrypt-native:x86_64-linux:1.10.1:r0:x86_64:10:a61860a7702fbf31e63bdcbadf5348983eecb402fb01a4de25a91d8b9e45a9a3_populate_sysroot.tar.zst.siginfo;downloadfilename=universal/a6/18/sstate:libgcrypt-native:x86_64-linux:1.10.1:r0:x86_64:10:a61860a7702fbf31e63bdcbadf5348983eecb402fb01a4de25a91d8b9e45a9a3_populate_sysroot.tar.zst.siginfo anywhere. The paths that were searched were: 2024-09-12T08:11:19.2793587Z /__w/1/s/DS10G/yocto/dgc220_release/dgc220/build/sstate-cache In the build above 5 other packages failed to fetch at the same time (within 50ms). The exact packages that fail and number of packages that fail vary from build to build. I have managed to capture a tcpdump on the docker host machine and the access logs of the NGINX server at the time of the above build. The access logs show a HEAD and GET pair to check the tar.zst exists and fetch it, then HEAD request for the tar.zst.siginfo which returns OK but no corresponding GET. About 16s later the build errors happen. I can see no obvious errors in the tcpdump and no sign of failed attempts to GET the siginfo. There are no errors reported from NGINX. >From reading other posts I accept this is probably some issue with our network >that needs to be sorted, but I don't have much to give our IT department to go >on other than it's not working reliably. Is there a way I could get yocto/bitbake to produce more logs or diagnostics as to why/how the fetch of the siginfo part of the sstate cache is failing (before the warning/error that the local file doesn't exist)? Or do you have any other advice that might help? Best Regards, Phil Dawson
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#63851): https://lists.yoctoproject.org/g/yocto/message/63851 Mute This Topic: https://lists.yoctoproject.org/mt/108501743/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
