On 03/07/2026 10:22, Matthew Brost wrote:
On Thu, Jul 02, 2026 at 03:37:37PM +0100, Tvrtko Ursulin wrote:
The problem statement is explained quite well and succinctly at:
https://gitlab.freedesktop.org/panfrost/linux/-/work_items/49
Essentially, on a system (over)loaded with a lot of runnable CPU processes, a
high-priority DRM client gets latency injected into the GPU submission path due
to the DRM scheduler use of workqueues.
This patch series proposes to replace the workqueues with kthread_work and
priority inheritance to solve this problem.
In the above linked issue Chia-I benchmarked the submit latencies which
show a striking improvement:
median 95% 99%
before 41us 1.5ms 2.6ms
after 15us 19us 24us
Can you give more information on these numbers? e.g., What you ran / how
you measured these. It is hard to argue with numbers.
I believe Chia-I observed latency on some production hw/sw and then
wrote a synthetic benchmark to test it more easily. Details are in the
above linked issue.
This is obviously really good for preventing compositors from missing frames and
Modern compositors do not pass render-job fences to draw jobs as input
dependencies. On Wayland, this functionality is provided by
linux-drm-syncobj-v1 and is enabled by default on Ubuntu 24.10 and
later. SurfaceFlinger has also operated this way for quite some time.
The obvious solution for compositors is to submit work directly through
the ioctl (i.e., bypass path in sched) when there are no input
dependencies, which should be the common case. The main exception is
when one of the BOs mapped into the compositor is migrating or otherwise
busy (i.e., a BO has a fence in a kernel dma-resv slot).
You mean the direct submit RFC you floated some time ago? What was the
verdict on that one, wasn't it rejected?
similar. Another good quote for the above issue, explaining the consequence of
CPU starvation of the submit path, is this:
"""
As a result, vkQueueWaitIdle blocks for 9.5ms for a gpu job that takes 4.5ms
gpu time.
"""
More details here?
This just illustrates how long can the workqueue wait for it's slice on
the CPU if the cores are loaded with other tasks. In other words, how
long since job is runnable to it actually being passed to the GPU.
DRM scheduler was originally using kthreads but was converted workqueues due
desire by xe to create thousands of schedulers. This series also questions
whether that was needed, given how the submission is serialized by a device
global lock (per GT, so almost device global). Panthor has a similar situation;
hence the series contains two patches to move those two to a setup which matches
the design of those drivers.
Other drivers, like for example amdgpu, v3d, etnaviv etc, which use the
scheduler as a hardware scheduler, where number of instances follow the number
of hardware blocks instead the number of userspace contexts, are completely
fine.
There are use cases however which do currently track the number of userspace
contexts and which do allow for more parallelism. For those a straight
kthread_work conversion would be a problem due an explosion in number of
threads.
The most direct example is panthor VM bind queue which creates a scheduler per
userspace context and relies on work queue concurrency management to keep the
number of threads in check.
This creates a challenge for the kthread_work conversion. To solve which I for
now opted to create a trivial round-robin thread pool. For the RFC this is
limited to four CPU threads and is something which will need to be discussed.
4 CPU threads per device, per drm sched module?
4 CPU threads per drm_sched_create_concurrent_worker(). So depends on
the driver how it wants to use it. For xe there are no usages of that,
albeit you say single thread is not workable, we can discuss that in the
other sub-thread. For panthor this RFC uses that flavour of the worker
for the VM bind queues and in that case it is 4 CPU threads per device
which handle VM bind requests from all userspace clients.
Regards,
Tvrtko
I have more questions but let's get some clarification first.
Matt
Ie. how much parallelsim those really need. The true answer is somewhere between
"at most the number of active userspace contexts and the number of CPU cores".
Or it could be less than that, since after all, VM BIND parallelism is
eventually going to choke on a narrower gate of actual GPU execution. We could
also allow drivers to pick their number.
In terms of how I implemented priority inheritance, the most important
characteristic is that it is temporary. As many userspace clients may be
submitting to a single DRM scheduler instance, a generic solution is to only
elevate the submission worker priority while there are active high priority
submitters. The mechanism is light weight and has a hysteresis built in to avoid
frequent scheduler operations.
That's pretty much it for now apart for an important detail that this RFC will
not build for all drivers! Out of those which directly use the DRM scheduler
APIs changed, I converted only panthor and xe. Amdgpu will also work by the way.
Others I have not tried to build.
Cc: Boris Brezillon <[email protected]>
Cc: Steven Price <[email protected]>
Cc: Liviu Dudau <[email protected]>
Cc: Chia-I Wu <[email protected]>
Cc: Danilo Krummrich <[email protected]>
Cc: Matthew Brost <[email protected]>
Cc: Philipp Stanner <[email protected]>
Tvrtko Ursulin (8):
drm/panthor: Remove redundant drm_sched_job_cleanup() from the
.free_job callback
drm/panthor: Use separate workqueue for DRM scheduler
drm/sched: Use generic naming for workqueue helpers
drm/xe: Convert to per gt scheduler workers
drm: Wrap DRM scheduler worker in own abstraction
drm/sched: Convert the scheduler job submission to kthread_worker
drm/sched: Add ability to change drm_sched_worker priority
drm/sched: Notify worker of the entity submission priority
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 8 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c | 4 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c | 2 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.c | 8 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.c | 8 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.c | 2 +-
drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c | 4 +-
drivers/gpu/drm/amd/amdgpu/vcn_v5_0_1.c | 4 +-
drivers/gpu/drm/msm/adreno/adreno_device.c | 4 +-
drivers/gpu/drm/panthor/panthor_mmu.c | 20 +-
drivers/gpu/drm/panthor/panthor_sched.c | 23 ++-
drivers/gpu/drm/scheduler/sched_entity.c | 3 +
drivers/gpu/drm/scheduler/sched_main.c | 168 +++++++++++-----
drivers/gpu/drm/scheduler/sched_rq.c | 202 ++++++++++++++++++++
drivers/gpu/drm/xe/xe_dep_scheduler.c | 6 +-
drivers/gpu/drm/xe/xe_dep_scheduler.h | 6 +-
drivers/gpu/drm/xe/xe_exec_queue.c | 6 +-
drivers/gpu/drm/xe/xe_gpu_scheduler.c | 21 +-
drivers/gpu/drm/xe/xe_gpu_scheduler.h | 2 +-
drivers/gpu/drm/xe/xe_gpu_scheduler_types.h | 2 +-
drivers/gpu/drm/xe/xe_gt.c | 7 +
drivers/gpu/drm/xe/xe_gt_types.h | 3 +
drivers/gpu/drm/xe/xe_guc_submit.c | 18 +-
drivers/gpu/drm/xe/xe_tlb_inval.c | 14 +-
drivers/gpu/drm/xe/xe_tlb_inval_types.h | 5 +-
include/drm/gpu_scheduler.h | 131 ++++++++++++-
27 files changed, 551 insertions(+), 134 deletions(-)
--
2.54.0
