On 03/07/2026 09:52, Philipp Stanner wrote:
+Cc Tejun
On Thu, 2026-07-02 at 15:37 +0100, Tvrtko Ursulin wrote:
[…]
DRM scheduler was originally using kthreads but was converted workqueues due
desire by xe to create thousands of schedulers. This series also questions
whether that was needed, given how the submission is serialized by a device
global lock (per GT, so almost device global). Panthor has a similar situation;
hence the series contains two patches to move those two to a setup which matches
the design of those drivers.
Other drivers, like for example amdgpu, v3d, etnaviv etc, which use the
scheduler as a hardware scheduler, where number of instances follow the number
of hardware blocks instead the number of userspace contexts, are completely
fine.
There are use cases however which do currently track the number of userspace
contexts and which do allow for more parallelism. For those a straight
kthread_work conversion would be a problem due an explosion in number of
threads.
The most direct example is panthor VM bind queue which creates a scheduler per
userspace context and relies on work queue concurrency management to keep the
number of threads in check.
This creates a challenge for the kthread_work conversion. To solve which I for
now opted to create a trivial round-robin thread pool. For the RFC this is
limited to four CPU threads and is something which will need to be discussed.
Ie. how much parallelsim those really need. The true answer is somewhere between
"at most the number of active userspace contexts and the number of CPU cores".
Or it could be less than that, since after all, VM BIND parallelism is
eventually going to choke on a narrower gate of actual GPU execution. We could
also allow drivers to pick their number.
Anyone remember whether that was discussed back in the day of
converting to workqueue?
AFAIR the only thing was that existing usage of kthreads did not scale
for the new use case of one DRM scheduler per userspace context.
What makes this idea suspicious in my mind is that the workqueue
implementation exists to solve precisely this issue: decide how many
kthreads really need to be spawned. AFAIK it can spawn additional
threads dynamically if necessary.
So the first question I would like to see considered is whether
workqueue could be improved in any way to address said latency issues.
Have you already considered that, Tvrtko?
Yes, priority inheritance was justifiably rejected for generic
workqueues. There is WQ_HIGHPRI, but the idea here is to go one step
further and allow tracking realtime scheduling policies.
IOW, is it the nature of workqueue or our way of using them in
drm_sched which causes this performance difference in the submit path?
Mostly self inflicted pain by the kthread->workqueue DRM scheduler
conversion.
Before we had threads which were woken up on first submission and would
fed the GPU until the software side queue was empty. With that design we
would still have the scheduling latency on the first submit but it would
be easy to add priority inheritance.
After the workqueue conversion, it is not only that we lost the ability
to do priority inheritance, but it was also changed to only feed one job
at the time to the GPU and rely on efficient worker re-queue.
It would seem that it's the desire for more fine-grained scheduling
characteristic control that you get with a kthread.
I think we have roughly three options:
1) Just use WQ_HIGHPRI and accept real time clients at best get re-nice
level boost. But no RT.
2) Do something along the lines of this RFC.
3) Split the scheduler into frontend and backend parts and make 1:1
drivers not use the actual scheduler.
The last one can be a combination of efforts. For example we can not go
for splitting the scheduler design, but adding a new dependency sorter
scheduler like Matthew was recently proposing. As long as that one can
handle priority inheritance ie. not suffer this submission latency issue
for 1:1 drivers it works.
M:N drivers we could then revert the code based back to kthreads and
probably simplify some things.
Downside is two schedulers in the codebase..
Regards,
Tvrtko
