Hi! On Wed, 2024-12-04 at 14:03:30 +0100, Guillem Jover wrote: > On Thu, 2024-11-28 at 10:54:37 +0100, Helmut Grohne wrote: > > Are there other layers that could reasonably be used to implement a more > > general form of parallelism limiting based on system RAM? Ideally, we'd > > consolidate these implementations into fewer places. > > I think adding this in dpkg-buildpackage itself would make most sense > to me, where it is already deciding what amount of parallelism to use > when specifying «auto» for example. > > Given that this would be and outside-in interface, I think this would > imply declaring these parameters say as debian/control fields for example, > or some other file to be parsed from the source tree. > > My main concerns would be: > > * Portability. > * Whether this is a local property of the package (so that the > maintainer has the needed information to decide on a value, or > whether this depends on the builder's setup, or perhaps both). > * We might need a way to percolate these parameters to children of > the build/test system (as Paul has mentioned), where some times > you cannot specify this directly in the parent. Setting some > standardize environment variables would seem sufficient I think, > but while all this seems kind of optional, this goes a bit into > reliance on dpkg-buildpackage being the only supported build > entry point. :)
Ah, and forgot to mention, that for example dpkg-deb (via libdpkg) already implements this kind of parallelism limiter based on system memory when compressing to xz. But in that case we are assisted by liblzma telling us the amount of memory expected to be used, so it makes it easier to clamp the parallelism based on that. Unfortunately I'm not sure, in general, we have this kind of information available, and my assumption is that in many cases we might end up deciding on clamping factors out of current observations based on current implementation details, that might need manual tracking and adjustment going on. Thanks, Guillem
