On 06/07/2026 20:36, Tejun Heo wrote:
Hello,
On Mon, Jul 06, 2026 at 01:20:49PM +0100, Tvrtko Ursulin wrote:
...
So the first question I would like to see considered is whether
workqueue could be improved in any way to address said latency issues.
Have you already considered that, Tvrtko?
Yes, priority inheritance was justifiably rejected for generic workqueues.
There is WQ_HIGHPRI, but the idea here is to go one step further and allow
tracking realtime scheduling policies.
If deciding at work item boundaries is granular enough, it's not
unconscionable to support RT worker pools. The main reason for resisting
that is because it can be too easy to abuse. A couple RT threads may solve
immediate problems but with a bunch of them you just don't have a working
scheduler in the system. In general, it's pretty suspicious if there's
intersection of "I need a bunch of worker threads in a flexible way" and "I
want RT".
In this case I think the key will be fleshing out with individual driver
owners how many workers their hardware actually needs. Setting the upper
bound to the number of userspace contexts, capped by the cmwq system
limit, sounds over the top for many GPUs as there isn't really any point
to feed a single GPU with that many CPU threads.
So far the feedback is I went too aggressive by reducing panthor to one
submission worker per device and xe to between one to two. Lets see if
there is a middle ground, and assuming there is, and that it is some
relatively small-ish number relative to the number of CPU cores, maybe
then we talk about can we have a RT worker pool of a limited size.
Or if drivers will insist they need cmwq managed bound for the number of
threads then we may need to go to plan B and either split the scheduler
for the two use cases, or write a new one for firmware drivers.
How big a hit is WQ_HIGHPRI vs. RT?
I can measure that tomorrow or so. But so far I am not very optimistic
about MIN_NICE ability to make a difference.
Today I repeated the benchmarking Chia-I did originally, with the tweak
of background load threads always being of normal priority and only
varying the userspace Vulkan submit thread between normal, nice -1 and
FIFO. Those three against three kernels - stock, this RFC capped to
never go above MIN_NICE for the kthread_worker, and this RFC as is (so
kthread_worker temporarily follows userspace FIFO).
The results are that only following to FIFO brings interesting gains.
STOCK --RFC Full RFC
. N RT . N RT . N RT
M 28 32 35 36 26 23 19 19 17
95% 211 537 69 276 186 426 151 284 30
98% 843 993 804 942 1277 1842 982 1028 34
STOCK = 7.1.0
--RFC = This RFC, but kthread_worker capped and MIN_NICE
Full RFC = RFC as is, kthread_worker goes up to FIFO
M = Median
. = Userspace submit thread SCHED_OTHER
N = -||= nice -1
RT = -||- FIFO 1
All numbers are latencies between job passed to DRM scheduler to DRM
scheduler passing it to the GPU. In micro seconds.
I'll report back with WQ_HIGHPRI results as soon as possible.
Regards,
Tvrtko
