When adding https://cdn.jsdelivr.net/yocto/sstate/all please remove any reference to sstate.yoctoproject.org from SSTATE_MIRRORS to ensure a proper test.
Thank you On Sat, Sep 23, 2023 at 12:51 PM Michael Halstead < mhalst...@linuxfoundation.org> wrote: > After some workshopping this morning we have a solution for larger files. > Almost every package is now available with CDN acceleration. For files over > 64MB the fetcher will need to follow a series of redirects to work > properly. Files under 64MB will be served fast from every region. Larger > files will be proxyed and cached on a smaller network of servers that can > handle them. Once cached they will be stored for up to a month. Because > sstate is immutable that means fast access after the first use in a region. > > Set SSTATE_MIRRORS > <https://docs.yoctoproject.org/ref-manual/variables.html#term-SSTATE_MIRRORS> > to include https://cdn.jsdelivr.net/yocto/sstate/all or > http://cdn.jsdelivr.net/yocto/sstate/all, test it out, and report back. > > If we want the system to perform as fast as possible we need to change > SSTATE to work with 64MB file segments but that's a discussion to have > after this solution is tested. > > > On Fri, Sep 22, 2023 at 12:03 PM Michael Halstead < > mhalst...@linuxfoundation.org> wrote: > >> Here are packages and size ranges that are not yet served via CDN. >> Hopefully testing on the smaller files is possible while we figure out >> a solution for these. >> >> assimp 75M 76M >> binutils 64M 100M >> boost 68M 148M >> ceres-solver 167M 182M >> cmake 157M 553M >> ffmpeg 68M 69M >> gcc 127M 681M >> gcc-cross-aarch64 64M 102M >> gcc-cross-arm 64M 72M >> gcc-cross-i686 65M 80M >> gcc-cross-mips 65M 69M >> gcc-cross-mips64 64M 69M >> gcc-cross-powerpc 64M 72M >> gcc-cross-x86_64 65M 104M >> gcc-crosssdk-aarch64-pokysdk-linux 65M 75M >> gcc-crosssdk-i686-oesdk-linux 65M 77M >> gcc-crosssdk-i686-pokysdk-linux 65M 79M >> gcc-crosssdk-i686-w64-mingw32 66M 78M >> gcc-crosssdk-x86_64-oesdk-linux 65M 78M >> gcc-crosssdk-x86_64-pokysdk-linux 65M 80M >> gcc-crosssdk-x86_64-w64-mingw32 67M 79M >> gdb 66M 105M >> ghostscript 64M 115M >> git 68M 90M >> glibc-locale 81M 228M >> go 67M 67M >> go-binary-native 82M 82M >> go-cross-core2-32 98M 100M >> go-cross-core2-64 93M 100M >> go-cross-corei7-64 98M 99M >> go-cross-cortexa15t2hf-neon 99M 99M >> go-cross-cortexa57 93M 99M >> go-cross-cortexa8hf-neon 99M 100M >> go-cross-mips32r2 98M 99M >> go-cross-mips64 99M 99M >> go-cross-mips64r2 99M 99M >> go-native 86M 86M >> go-runtime 64M 82M >> grpc 74M 102M >> hdf5 78M 137M >> influxdb 76M 98M >> intel-media-driver 68M 132M >> jemalloc 164M 164M >> kea 65M 153M >> lib32-binutils 64M 70M >> lib32-boost 77M 122M >> lib32-cmake 158M 305M >> lib32-gcc 144M 390M >> lib32-gcc-cross-i686 65M 79M >> lib32-gcc-cross-mips 65M 69M >> lib32-ghostscript 65M 83M >> lib32-git 65M 65M >> lib32-glibc-locale 81M 228M >> lib32-go-cross-x86 98M 99M >> lib32-kea 64M 88M >> lib32-linux-firmware 275M 419M >> lib32-llvm 1126.4M 2662.4M >> lib32-ltp 77M 305M >> lib32-mesa 65M 68M >> lib32-openssl 119M 360M >> lib32-piglit 106M 106M >> lib32-qemu 275M 639M >> lib32-rust 93M 129M >> lib32-rust-llvm 64M 76M >> lib32-spirv-tools 68M 107M >> lib32-valgrind 68M 89M >> lib32-vulkan-demos 74M 74M >> lib32-vulkan-validation-layers 65M 101M >> lib32-webkitgtk 1126.4M 927M >> lib64-boost 75M 119M >> lib64-gcc 151M 401M >> lib64-gcc-cross-mips64 65M 69M >> lib64-gcc-cross-x86_64 65M 80M >> lib64-glibc-locale 82M 228M >> lib64-go-cross-x86_64 98M 99M >> lib64-go-runtime 68M 81M >> lib64-ltp 67M 347M >> lib64-mesa 66M 68M >> lib64-openssl 276M 411M >> lib64-rust 97M 123M >> lib64-rust-llvm 64M 67M >> lib64-valgrind 65M 84M >> libyang 93M 93M >> linux-firmware 275M 419M >> linux-intel 120M 395M >> linux-yocto 66M 537M >> linux-yocto-rt 174M 287M >> llvm 1126.4M 3481.6M >> llvm-native 71M 88M >> ltp 64M 598M >> mariadb 1331.2M 966M >> mariadb-native 237M 259M >> mesa 64M 113M >> minifi-cpp 155M 214M >> mozjs-102 183M 255M >> nativesdk-glibc-locale 81M 226M >> nativesdk-qemu 245M 644M >> nativesdk-rust 82M 125M >> nativesdk-rust-llvm 65M 73M >> nodejs 1024.0M 723M >> opencv 185M 237M >> opengl-es-cts 406M 560M >> openssl 92M 685M >> php 67M 68M >> piglit 106M 106M >> poco 75M 107M >> python3-grpcio 65M 88M >> python3-wxgtk4 70M 100M >> qemu 1228.8M 929M >> qemu-system-native 65M 71M >> renderdoc 67M 82M >> rocksdb 177M 272M >> rust 85M 140M >> rust-llvm 64M 79M >> rust-llvm-native 287M 418M >> rust-native 78M 85M >> spirv-tools 64M 195M >> systemd 69M 71M >> tesseract-lang 369M 451M >> upm 82M 82M >> valgrind 64M 128M >> vsomeip 82M 149M >> vulkan-cts 592M 833M >> vulkan-demos 73M 138M >> vulkan-validation-layers 65M 108M >> webkitgtk 1024.0M 995M >> webkitgtk3 1126.4M 1638.4M >> wireshark 65M 85M >> wxwidgets 70M 88M >> >> On Fri, Sep 22, 2023 at 10:32 AM Michael Halstead < >> mhalst...@linuxfoundation.org> wrote: >> >>> Right now we have https://cdn.jsdelivr.net/yocto/sstate set up as a CDN >>> mirroring proxy endpoint. It's fast and completely free to the project. >>> However, it will only work for files smaller than 64MB and will send a >>> redirect to the main server at sstate.yoctoproject.org for files larger >>> than 64MB. >>> >>> The good news is that we have a fast free CDN provided by >>> https://www.jsdelivr.com/ and their many sponsors for 99.4% of our >>> files. >>> >>> Here is an estimated distribution of sstate files currently available: >>> 1 KB-4 KB: 103168 >>> 4 KB-16 KB: 1218048 >>> 16 KB-64 KB: 1886976 >>> 64 KB-256 KB: 1225216 >>> 256 KB-1 MB: 170752 >>> 1 MB-4 MB: 92928 >>> 4 MB-16 MB: 83200 >>> 16 MB-64 MB: 40448 >>> 64 MB-256 MB: 17664 >>> 256 MB-1 GB: 7936 >>> 1 GB-4 GB: 2560 >>> >>> I'm still working on a CDN solution for the larger files but we can >>> begin testing https://cdn.jsdelivr.net/yocto/sstate today. >>> >>> Here is an example link to a file that is currently popular from each >>> location: >>> >>> >>> http://sstate.yoctoproject.org/all/universal/c6/81/sstate:re2c-native:x86_64-linux:3.1:r0:x86_64:11:c681f19e2786732c7920860d0b27cdcf3a79c9441d9bdaa82319a12e96e6246e_deploy_source_date_epoch.tar.zst >>> >>> >>> https://cdn.jsdelivr.net/yocto/sstate/all/universal/c6/81/sstate:re2c-native:x86_64-linux:3.1:r0:x86_64:11:c681f19e2786732c7920860d0b27cdcf3a79c9441d9bdaa82319a12e96e6246e_deploy_source_date_epoch.tar.zst >>> >>> I'm also attaching a list of the largest files in sstate. Anything we >>> can do to reduce the size of these or to split them into smaller pieces >>> will help with hosting and with the user's download experience. >>> >>> -- >>> Michael Halstead >>> Linux Foundation / Yocto Project >>> Systems Operations Engineer >>> >> >> >> -- >> Michael Halstead >> Linux Foundation / Yocto Project >> Systems Operations Engineer >> > > > -- > Michael Halstead > Linux Foundation / Yocto Project > Systems Operations Engineer > -- Michael Halstead Linux Foundation / Yocto Project Systems Operations Engineer
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61088): https://lists.yoctoproject.org/g/yocto/message/61088 Mute This Topic: https://lists.yoctoproject.org/mt/101525879/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/leave/6691583/21656/737036229/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
