Some CDN stats are now published at https://www.jsdelivr.com/oss-cdn/yocto. Let me know if you notice anything that needs to be changed.
On Wed, Oct 18, 2023 at 10:33 AM Richard Purdie < richard.pur...@linuxfoundation.org> wrote: > On Wed, 2023-10-18 at 14:02 +0200, Alexander Kanavin wrote: > > On Wed, 18 Oct 2023 at 08:45, Richard Purdie > > <richard.pur...@linuxfoundation.org> wrote: > > > I'm torn on the targets to test as sato-sdk is a large image and world > > > is a lot of work. I'd be tempted to test sato, weston and full-cmdline? > > > World is a good test I guess and if from sstate, shouldn't have that > > > much of an issue. It does also prove things are working. > > > > I ran '-S printdiff world' on a blank build directory. First, > > scalability isn't great: > > > > Initialising tasks: 100% > > > > ##########################################################################################################################################################################| > > Time: 0:24:19 > > Checking sstate mirror object availability: 100% > > > > ##################################################################################################################################################| > > Time: 0:12:14 > > > > So it's taking 36 minutes just preparing to fetch the objects, and 2/3 > > of that time goes into communicating with hash equivalence server > > (e.g. BB_HASHSERVE_UPSTREAM = "hashserv.yocto.io:8687"). > > FWIW the second time an artefact is pulled from the CDN, it will be > much faster. I don't know how much that is a factor here. > > We know hashequiv is slow, particularly from europe. We've ideas on > that we're looking into but for now it is what it is. Part of this work > was to find out what the issues are so we write that one up. > > > Second, there are significant misses. I don't have a clear theory > > where they come from, just want to list them: > > > > The differences between the current build and any cached tasks start > > at the following tasks: > > /srv/work/alex/poky/meta/recipes-core/meta/meta-ide-support.bb: > do_populate_ide_support > > /srv/work/alex/poky/meta/recipes-graphics/xorg-driver/ > xf86-input-synaptics_1.9.2.bb:do_collect_spdx_deps > > > /srv/work/alex/poky/meta/recipes-devtools/bootchart2/bootchart2_0.14.9.bb: > do_create_runtime_spdx > > > /srv/work/alex/poky/meta/recipes-graphics/igt-gpu-tools/igt-gpu-tools_git.bb: > do_collect_spdx_deps > > > /srv/work/alex/poky/meta/recipes-devtools/python/python3-dbusmock_0.29.1.bb: > do_create_runtime_spdx > > /srv/work/alex/poky/meta/recipes-devtools/dpkg/dpkg_1.22.0.bb: > do_configure > > > /srv/work/alex/poky/meta/recipes-graphics/xorg-driver/xf86-input-vmmouse_13.2.0.bb: > do_collect_spdx_deps > > /srv/work/alex/poky/meta/recipes-graphics/glew/glew_2.2.0.bb: > do_collect_spdx_deps > > > /srv/work/alex/poky/meta/recipes-graphics/xorg-driver/xf86-input-evdev_2.10.6.bb: > do_collect_spdx_deps > > /srv/work/alex/poky/meta/recipes-graphics/libva/libva_2.19.0.bb: > do_collect_spdx_deps > > /srv/work/alex/poky/meta/recipes-sato/webkit/libwpe_1.14.1.bb: > do_collect_spdx_deps > > > /srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gstreamer1.0-meta-base.bb: > do_collect_spdx_deps > > /srv/work/alex/poky/meta/recipes-multimedia/gstreamer/ > gstreamer1.0-python_1.22.6.bb:do_collect_spdx_deps > > /srv/work/alex/poky/meta/recipes-extended/cups/cups_2.4.6.bb: > do_collect_spdx_deps > > /srv/work/alex/poky/meta/recipes-gnome/libdazzle/libdazzle_3.44.0.bb: > do_collect_spdx_deps > > > /srv/work/alex/poky/meta/recipes-graphics/igt-gpu-tools/igt-gpu-tools_git.bb: > do_package > > > /srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gst-devtools_1.22.6.bb: > do_collect_spdx_deps > > > /srv/work/alex/poky/meta/recipes-core/packagegroups/packagegroup-core-sdk.bb: > do_package > > > /srv/work/alex/poky/meta/recipes-graphics/xorg-driver/xf86-input-mouse_1.9.5.bb: > do_collect_spdx_deps > > /srv/work/alex/poky/meta/recipes-graphics/waffle/waffle_1.7.2.bb: > do_collect_spdx_deps > > /srv/work/alex/poky/meta/recipes-multimedia/alsa/alsa-tools_1.2.5.bb: > do_collect_spdx_deps > > > /srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gstreamer1.0-rtsp-server_1.22.6.bb: > do_collect_spdx_deps > > /srv/work/alex/poky/meta/recipes-devtools/apt/apt_2.6.1.bb:do_install > > /srv/work/alex/poky/meta/recipes-gnome/gtk+/gtk4_4.12.3.bb: > do_collect_spdx_deps > > /srv/work/alex/poky/meta/recipes-devtools/devel-config/distcc-config.bb: > do_create_runtime_spdx > > /srv/work/alex/poky/meta/recipes-gnome/libhandy/libhandy_1.8.2.bb: > do_collect_spdx_deps > > /srv/work/alex/poky/meta/recipes-support/vim/vim_9.0.bb: > do_collect_spdx_deps > > We'll have to dig into this. > > > So I think we should start with specific images first - minimal, > > full-cmdline, weston, sato and sato-sdk are all much faster to check. > > Agreed, lets start with simpler images, full-cmdline, minimal, sato and > weston. We can increase coverage as we move forward (world, sato-sdk). > > > On qemux86_64 none of them show misses, but on qemuarm64 there are > > problems with sato, sato-sdk and weston, i.e. sato-sdk shows: > > > > The differences between the current build and any cached tasks start > > at the following tasks: > > /srv/work/alex/poky/meta/recipes-gnome/gdk-pixbuf/gdk-pixbuf_2.42.10.bb: > do_package_write_rpm > > > /srv/work/alex/poky/meta/recipes-connectivity/connman/connman-gnome_0.7.bb: > do_package_write_rpm > > > /srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gst-examples_1.18.6.bb: > do_package_write_rpm > > /srv/work/alex/poky/meta/recipes-sato/images/core-image-sato-sdk.bb: > do_deploy_source_date_epoch > > /srv/work/alex/poky/meta/recipes-graphics/xorg-font/font-util_1.4.1.bb: > do_package_write_rpm > > > /srv/work/alex/poky/meta/recipes-core/packagegroups/packagegroup-core-sdk.bb: > do_package > > /srv/work/alex/poky/meta/recipes-gnome/librsvg/librsvg_2.56.3.bb: > do_package_write_rpm > > > /srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-bad_1.22.6.bb: > do_package_write_rpm > > > > > > I'm not sure if that's because cache for the current master needs to > > be populated properly first, or if there's a deeper issue. > > We made a release build with current master so the cache should be > good. We're going to have to look into it. > > > > Either we start logging what we've built so we get the last known > > > revisions, or we run the test as part of a-quick/a-full at the end of > > > the build? I don't really want to extend the build but I'm not sure we > > > may have much choice. > > > > I think it can be a selftest and bundled into > > trigger-build-posttrigger which already runs > > 'buildoptions.SourceMirroring.test_yocto_source_mirror': > > > > > https://git.yoctoproject.org/yocto-autobuilder-helper/tree/config.json#n465 > > > > The various yocto mirrors tests could be tagged in a way similar to > > machine/toolchain selftests, so that they're excluded from regular > > oe-selftest, but run specifically here. > > That sounds reasonable. oe-selftest has the concept of "tags" so we > should create some for these tests and use those as the hardcoded lists > are a pain. We already do that for the toolchain-user, toolchain-system > and machine ones iirc. > > Cheers, > > Richard > > > > > > -- Michael Halstead Linux Foundation / Yocto Project Systems Operations Engineer
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61559): https://lists.yoctoproject.org/g/yocto/message/61559 Mute This Topic: https://lists.yoctoproject.org/mt/101525879/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/leave/6691583/21656/737036229/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
