On 09/07/2026 09:28, Tvrtko Ursulin wrote:
On 08/07/2026 17:24, Tvrtko Ursulin wrote:
8<>
How big a hit is WQ_HIGHPRI vs. RT?
I can measure that tomorrow or so. But so far I am not very optimistic
about MIN_NICE ability to make a difference.
Today I repeated the benchmarking Chia-I did originally, with the
tweak of background load threads always being of normal priority and
only varying the userspace Vulkan submit thread between normal, nice
-1 and FIFO. Those three against three kernels - stock, this RFC
capped to never go above MIN_NICE for the kthread_worker, and this RFC
as is (so kthread_worker temporarily follows userspace FIFO).
The results are that only following to FIFO brings interesting gains.
STOCK --RFC Full RFC
. N RT . N RT . N RT
M 28 32 35 36 26 23 19 19 17
95% 211 537 69 276 186 426 151 284 30
98% 843 993 804 942 1277 1842 982 1028 34
STOCK = 7.1.0
--RFC = This RFC, but kthread_worker capped and MIN_NICE
Full RFC = RFC as is, kthread_worker goes up to FIFO
M = Median
. = Userspace submit thread SCHED_OTHER
N = -||= nice -1
RT = -||- FIFO 1
All numbers are latencies between job passed to DRM scheduler to DRM
scheduler passing it to the GPU. In micro seconds.
I'll report back with WQ_HIGHPRI results as soon as possible.
I made panthor set WQ_HIGHPRI on stock 7.1 and tested normal, nice and
FIFO vulkan submit thread. As expected I do not see a clear improvement:
. N RT
M 27 28 32
95% 163 246 809
98% 924 991 1882
This makes it seem that the 95-th percentile is perhaps better for
normal and re-niced userspace thread cases, but it may also be noise and
I would need to do many more and longer runs on each config to be sure.
98-th percentile spikes are still there. Also, even if 95% would be a
real improvement they are still an order of magnitude above median so I
do not think it is worth spending much more time on benchmarking that
option.
To add another data point, today I did a quick WQ_RTPRI experiment.
That allows eliminating both the kthread_worker pool complications and
priority inheritance from this series by allowing creation of SCHED_FIFO
workqueues. It reduces the series to just two patches. One to add
WQ_RTPRI and second to make panthor use it for realtime queues.
In the core code I capped max_active for those to num_online_cpus() / 2,
although I am not sure Tejun if that is what you had in mind when you
said it might be acceptable if the number of workers would be bound? I
am also not sure if this will limit the system number or just for
individual worker. If only the latter than further improvements will
obviously be required.
On top of that, for the panthor use case we agreed that two threads is
good enough so that is what the driver requests as the upper bound.
Overall flow is that when userspace requests
VK_QUEUE_GLOBAL_PRIORITY_REALTIME (for which CAP_SYS_NICE is required
btw), that queue goes to the DRM scheduler panthor created a WQ_RTPRI
workqueue for. Lower priority Vulkan queues go to lower priority
workqueues so there is no danger of normal priority clients over-using
the RT queues.
Submit latency wise this is still completely effective. Median latency
is 14us and 95 and 98-th percentiles are 23us and 25us respectively.
So the question is whether something like this is acceptable to
workqueue core? If so, I can send it as a proper RFC and go from there.
Regards,
Tvrtko
P.S.
My WIP branch implementing this:
https://cgit.freedesktop.org/~tursulin/drm-intel/log/?h=wqrt-panthor
