Met vriendelijke groet / kind regards,
Mike Looijmans
System Expert
TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands
T: +31 (0) 499 33 69 69
E: [email protected]
W: www.topic.nl
Please consider the environment before printing this e-mail
On 11-04-2021 17:23, Gmane Admin via lists.yoctoproject.org wrote:
My build machine has 8 cores + HT so bitbake enthusiastically assumes
16. However I have (only?) 16GB of RAM (+24GB swap space).
Been there, done that (8/16 cores with 8GB RAM).
The major cause is that bitbake will not just spawn 16 compiler threads,
it will actually spawn up to 16 "do_compile" tasks which may spawn 16
compiler processes each, thus you'll be running a whopping 256
compilers. Bitbake may spawn a total of n^2 threads by default with "n"
the detected number of cores.
The workaround I used was to limit the number of bitbake threads but
leave the make threads at 16. Since most software these days is using
parallel make, the impact on the build time of reducing the bb threads
to 8 or 4 is negligible.
As for translating this into a bitbake feature, an implementation that
comes to mind is to add a "weight" to each task. Most tasks would get a
weight of just "1", but a (multithreaded) compile would be weighted "8"
(some formula involving the number of CPUs) or so. This would stop
bitbake spawning more processes early so that you don't get into the n^2
threads problem, and the number of processed that bitbake will spawn
will be close to linear agin instead of quadratic as is is now.
Recipes that have excessive memory loads (usually C++ template
meta-programming) can then just increase the weighting for the compile task.
--
Mike Looijmans
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#53077): https://lists.yoctoproject.org/g/yocto/message/53077
Mute This Topic: https://lists.yoctoproject.org/mt/82015730/21656
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
