Hi Michael and Pádraig, On Wed, Dec 25, 2024 at 06:59:28PM +0000, Pádraig Brady wrote: > On 25/12/2024 15:24, Michael Stone wrote: > > There's zero chance I'll carry this as a debian-specific fork of nproc. > > (Because I don't want to carry any new forks of the core utilities as > > doing so inevitably causes long-term pain.) If you can convince > > upstream, that's fine. My personal feeling is that nproc isn't the right > > place for it but I'll defer to upstream[1]. (Among other things I don't > > understand why this is better than extending the debian tooling in this > > area, and there's nothing but an assertion that doing so is bad; I'd > > suggest expanding on that.) > > > > [1] My gut says that what you really need is a different tool to show > > the memory available to the current process (which may not be the same > > as the amount of free memory on the system, e.g., in the presence of > > control groups). You could then divide that number by the expected > > memory per process and set your parallelization factor to the lesser of > > that value or nprocs. Conflating "nproc" and "nmem" seems wrong.
I note that I intentionally left cgroups out of the picture for now. Supporting cgroups does not further affect the command line flags (as it merely adds precision) and my earlier proposal for debhelper did consider cgroups. It is something I see me add in later iterations iff we agree on moving forward with this. > With my upstream hat on, coupling memory details in nproc > seems like not a good fit and better done outside that tool. Thank you both for your quick feedback on the proposed solution. I did not expect my patch to be taken as is, but the presence of a patch helped clarifying what I intended here. If you are reasonably sure that my patch will be rejected upstream, please tag this bug wontfix and close it. Otherwise, I may try going there. Before we move on, please allow me to ask what problem the nproc tool is supposed to solve. Of course it tells you the number of processors, but that is not a use case on its own. If you search for uses, "make -j$(nproc)" (with varying build systems) is the immediate hit. This use is now broken by hardware trends (see below). It can also be used to partition a system and computing partition sizes. This use case continues to work. Are there other use cases? Unless I am missing something, it seems like nproc no longer solves its most common use case (and this bug is asking to fix it). Thus far, I am effectively turned down by: * coreutils * debhelper * dpkg What other place would be suitable for including this functionality? We have a pattern of packages coming up with code chewing /proc/meminfo using various means (refer to my initial mail referenced from the bug submission) and reducing parallelism based on it. You may ask why there is a need for change here. I see a hardware trend wherein the supply in CPU cores grows faster than the supply in RAM at a time where the demand for RAM (e.g. in linker processes) grows. As a result, I expect more and more packages to face this need and want to supply convenient tooling such that we can pass failure to build a package in an environment with limited RAM onto the package rather than the build environment. This will rightfully evoke resistance from maintainers unless there is sensible tooling, which is what I am trying to solve in this discussion and bug report. Do you see the computation of allocatable RAM as something we can accommodate in coreutils? Michael suggested adding "nmem" between the lines. Did you mean that in an ironic way or are you open to adding such a tool? It would solve a quite platform-dependent part of the problem and significantly reduce the boiler plate in real use cases. So my search for a suitable spot to solve this continues. Help appreciated. Helmut
