Hi Guillem and other developers,
I am one of those who builds a lot of different packages with different
requirements and found that picking a good parallel=... value in
DEB_BUILD_OPTIONS is hard. Go too low and your build takes very long. Go
too high and you swap until the OOM killer terminates your build. (Usage
of choom recommended in any case.)
Common offenders are rustc and llvm-toolchain, but there are a few more.
These packages already offer some assistance.
llvm-toolchain-19 parses /proc/meminfo and reduces parallelism:
https://sources.debian.org/src/llvm-toolchain-19/1%3A19.1.4-1/debian/rules/#L92
A number of packages use debhelper's --max-parallel:
https://codesearch.debian.net/search?q=--max-parallel%3D%5B%5E1%5D+path%3Adebian%2Frules&literal=0
polymake limits parallelity to 4GB per core:
https://sources.debian.org/src/polymake/4.12-3/debian/rules/?hl=15#L15
Another implementation is in openscad:
https://sources.debian.org/src/openscad/2021.01-8/debian/rules/?hl=33#L33
ns3 turns off parallelism if the RAM is too limited:
https://sources.debian.org/src/ns3/3.42-2/debian/rules/?hl=26#L26
I think this demonstrates that we probably have something between 10 and
50 packages in unstable that would benefit from a generic parallelism
limit based on available RAM. Do others agree that this is a problem
worth solving in a more general way?
For one thing, I propose extending debhelper to provide
--min-ram-per-parallel-core as that seems to be the most common way to
do it. I've proposed
https://salsa.debian.org/debian/debhelper/-/merge_requests/128
to this end.
Unfortunately, a the affeted packages tend to not just be big, but also
so special that they cannot use dh_auto_*. As a result, I also looked at
another layer to support this and found /usr/share/dpkg/buildopts.mk,
which sets DEB_BUILD_OPTION_PARALLEL by parsing DEB_BUILD_OPTIONS. How
about extending this file with a mechanism to reduce parallelity? I am
attaching a possible extension to it to this mail to see what you think.
Guillem, is that something you consider including in dpkg?
Are there other layers that could reasonably be used to implement a more
general form of parallelism limiting based on system RAM? Ideally, we'd
consolidate these implementations into fewer places.
As I am operating build daemons (outside Debian), I note that I have to
limit their cores below what is actually is available to avoid OOM
kills and even that is insufficient in some cases. In adopting such a
mechanism, we could generally raise the core count per buildd and
consider OOM a problem of the package to be fixed by applying a sensible
parallelism limit.
Helmut
# Utilities
dpkg_buildopts_empty ?=
dpkg_buildopts_space ?= $(dpkg_buildopts_empty) $(dpkg_buildopts_empty)
dpkg_buildopts_max ?= $(shell test '$(1)' -ge '$(2)' && echo '$(1)' || echo
'$(2)')
dpkg_buildopts_min ?= $(shell test '$(1)' -le '$(2)' && echo '$(1)' || echo
'$(2)')
dpkg_buildopts_arith ?= $(shell echo "$$(($(1)))")
ifneq (,$(DEB_BUILD_OPTION_PARALLEL))
# Cap parallelism to DPKG_BUILDOPTS_MAX_PARALLEL
ifneq (,$(DPKG_BUILDOPTS_MAX_PARALLEL))
DEB_BUILD_OPTION_PARALLEL := $(call
dpkg_buildopts_min,$(DEB_BUILD_OPTION_PARALLEL),$(DPKG_BUILDOPTS_MAX_PARALLEL))
endif
# Cap parallelism using DPKG_BUILDOPTS_MIN_RAM_PER_PARALLEL_CORE
ifneq (,$(DPKG_BUILDOPTS_MIN_RAM_PER_PARALLEL_CORE))
# Obtain the MemTotal value from /proc/meminfo in kB
dpkg_buildopts_memtotal := $(strip $(subst
:,$(dpkg_buildopts_space),$(patsubst MemTotal:%:kB,%,$(filter
MemTotal:%,$(subst $(dpkg_buildopts_space),:,$(file </proc/meminfo))))))
dpkg_buildopts_memtotal_mb := $(call
dpkg_buildopts_arith,$(dpkg_buildopts_memtotal) / 1024)
# Discover the current cgroup. Empty on non-Linux.
dpkg_buildopts_cgroup := $(lastword $(subst
:,$(dpkg_buildopts_space),$(file </proc/self/cgroup)))
# If a cgroup was determined, attempt to look up its memory limit.
ifneq (,$(dpkg_buildopts_cgroup))
# This value is in bytes
dpkg_buildopts_memcg_max := $(file
</sys/fs/cgroup/$(dpkg_buildopts_cgroup)/memory.max)
endif
# If there is a memory limit, reduce memtotal.
ifneq (,$(filter-out max,$(dpkg_buildopts_memcg_max)))
dpkg_buildopts_memcg_max_mb := $(call
dpkg_buildopts_arith,$(dpkg_buildopts_memcg_max) / 1024 / 1024)
dpkg_buildopts_memtotal_mb := $(call
dpkg_buildopts_min,$(dpkg_buildopts_memtotal_mb),$(dpkg_buildopts_memcg_max_mb))
endif
dpkg_buildopts_parallel_limit_ram := $(call dpkg_buildopts_max,1,$(call
dpkg_buildopts_arith,$(dpkg_buildopts_memtotal_mb) /
$(DPKG_BUILDOPTS_MIN_RAM_PER_PARALLEL_CORE)))
DEB_BUILD_OPTION_PARALLEL := $(call
dpkg_buildopts_min,$(DEB_BUILD_OPTION_PARALLEL),$(dpkg_buildopts_parallel_limit_ram))
endif
endif
