I'm not sure this is right as a global change. This fixes situations where 3 xz instances run on your specific setup, what about 4? Or 5? Or n instances for all n in N?
I think you need to limit the amount of threads rather to match available RAM, and do it via site.conf and not bitbake.conf. ?= assignment is used in bitbake.conf exactly for that purpose. Alex On Sat, 27 Aug 2022 at 20:15, Edwin Török via lists.openembedded.org <edwin=etorok....@lists.openembedded.org> wrote: > > By default a build such as [1] might run 3 'xz' in parallel: > ``` > Currently 3 running tasks (11878 of 11883) 99% > |###################################################################### > #### | > 0: demo-coreip-xfce4-1.0-r0 do_image_ext4 - 3m17s (pid 2088739) > 1: demo-coreip-xfce4-1.0-r0 do_image_tar - 3m16s (pid 2088743) > 2: demo-coreip-xfce4-1.0-r0 do_image_wic - 3m16s (pid 2088745) > ``` > > However the default memory usage limit of `xz` is 50% each, so this > will > attempt to use 150% memory, and it gets OOM killed by systemd-oomd on > Fedora 36. > ``` > Aug 27 18:38:57 fedora systemd-oomd[3150]: Killed > /user.slice/user-1000.slice/user@1000.service/app.slice/app- > org.gnome.Terminal.slice/vte-spawn-2d92eb7b-b005-41b4-a786- > fc8c0d360ce3.scope due to memory used (66890584064) / total > (67332812800) and swap used (7744446464) / total (8589930496) being > more than 90.00% > ``` > > Even with systemd-oomd turned off it'd eventually start swapping > heavily > on a system with 64GiB of physical memory and 8GiB of swap. > > Reduce memory limit on xz so that we can run 3 in parallel without > driving the host close to or OOM. 25% seems to work on this particular > build and allows it to complete successfully. > > [1] https://github.com/sifive/freedom-u-sdk/tree/2022.06.00 > > Signed-off-by: Edwin Török <ed...@etorok.net> > --- > meta/conf/bitbake.conf | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index 2a3cf6f8aa..48ba52c12c 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -857,7 +857,8 @@ BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}" > PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" > > # Default parallelism and resource usage for xz > -XZ_MEMLIMIT ?= "50%" > +# A build might run 3 'xz' in parallel, so don't exhaust memory > +XZ_MEMLIMIT ?= "25%" > XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > XZ_THREADS[vardepvalue] = "1" > XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}" > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#169940): https://lists.openembedded.org/g/openembedded-core/message/169940 Mute This Topic: https://lists.openembedded.org/mt/93294351/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
