Hi Frederic, Thanks for having a crack at this, but I'm pretty confused now so please prepare for a bunch of silly questions!
On Tue, Oct 15, 2024 at 03:48:55PM +0200, Frederic Weisbecker wrote: > Le Tue, Oct 08, 2024 at 11:54:35AM +0100, Will Deacon a écrit : > > On Fri, Sep 27, 2024 at 12:48:59AM +0200, Frederic Weisbecker wrote: > > > When a kthread or any other task has an affinity mask that is fully > > > offline or unallowed, the scheduler reaffines the task to all possible > > > CPUs as a last resort. > > > > > > This default decision doesn't mix up very well with nohz_full CPUs that > > > are part of the possible cpumask but don't want to be disturbed by > > > unbound kthreads or even detached pinned user tasks. > > > > > > Make the fallback affinity setting aware of nohz_full. This applies to > > > all architectures supporting nohz_full except arm32. However this > > > architecture that overrides the task possible mask is unlikely to be > > > willing to integrate new development. > > > > I'm not sure I understand this last sentence. The possible mask is > > overridden for 32-bit tasks on an *arm64* kernel when running on an SoC > > featuring some CPUs that can execute only 64-bit tasks. Who is unwilling > > to integrate what? I should've been clearer in my reply, but I think the most important thing here for the arm64 heterogeneous SoCs is that we document whatever the behaviour is in Documentation/arch/arm64/asymmetric-32bit.rst. There are a few other kernel features that don't play well (e.g. SCHED_DEADLINE), so it might be sufficient just to call out the limitations relating to CPU isolation there. However: > Will, how does the (untested) following look like? The rationale is that > we must deal with the fact that CPU supporting 32-bits el0 may appear at > any time and those may not intersect housekeeping CPUs (housekeeping CPUs > are CPUs that are not part of nohz_full=. In the funky SoCs, all CPUs support 64-bit and we have a 64-bit kernel. Some CPUs additionally support 32-bit but that should only be a concern for the scheduling of user tasks. > If nohz_full= isn't used then > it's cpu_possible_mask). If there is a housekeeping CPU supporting el0 32bits > then it will be disallowed to be ever offlined. But if the first mismatching > CPU supporting el0 that pops up is not housekeeping then we may end up > with that CPU disallowed to be offlined + later if a housekeeping CPU appears > that also supports 32bits el0 will also be disallowed to be offlined. Ideally > it should turn back the previous CPU to be offlinable but there may be > other things that have forbidden that CPU to be offline so... I'd have thought the bigger problem would be if the set of nohz_full= CPUs was defined as the set of CPUs that support 32-bit. In that case, executing a 32-bit task will give the scheduler no choice but to run the task on a !housekeeping core. So perhaps we could turn this on its head and explicitly mark the first 32-bit capable CPU as a housekeeping core when the mismatched mode is enabled? We're already preventing CPU hotplug for the thing, so it's "special" already. If that conflicts with the nohz_full_option, we can emit a warning message that we're overriding it. I think that's ok, as the user will have had to specify 'allow_mismatched_32bit_el0' as well. Will
