On Mon, Nov 02, 2026 at 11:41:13AM +0100, Thomas Hellström wrote: > On Fri, 2026-10-30 at 14:16 -0700, Matthew Brost wrote: > > On Fri, Oct 30, 2026 at 04:17:20PM +0100, Thomas Hellström wrote: > > > On Tue, 2026-10-27 at 15:49 -0700, Matthew Brost wrote: > > > > Backing up a BO costs a GPU blit plus a page copy, and if the BO > > > > is > > > > part > > > > of an active working set it is faulted straight back in on the > > > > next > > > > exec. > > > > Evicting a BO that was used moments ago is therefore almost > > > > always a > > > > loss, yet the shrinker has no notion of recency beyond LRU order. > > > > > > > > Timestamp BOs wherever their LRU position is updated, and read > > > > the > > > > stamp > > > > back from whatever owns that position. A BO in a VM's bulk move > > > > is > > > > anchored inside its range, so it is aged by the VM-wide stamp > > > > taken > > > > in > > > > xe_vm_bump_lru() alongside the bulk move itself. Any other BO > > > > carries > > > > its own, stamped there too and on a successful move into TT in > > > > xe_bo_move(), where ttm_resource_alloc() has already placed the > > > > new > > > > resource at the LRU tail. > > > > > > > > Add a per-device shrinker_min_age_ms sysfs attribute, defaulting > > > > to > > > > CONFIG_DRM_XE_SHRINKER_MIN_AGE_MS (0, disabled). While set, the > > > > shrinker > > > > won't back up BOs younger than the delay. Each LRU priority is > > > > roughly > > > > age ordered, so the gate is applied per priority: > > > > xe_shrinker_count() > > > > snoops the first shrinkable BO of each priority and reports > > > > nothing > > > > only > > > > if all of them are young, and the backup walk skips to the next > > > > priority > > > > at the first young BO, both as a backstop for the count sample > > > > and so > > > > a > > > > hot priority doesn't shield colder BOs in another one. > > > > > > > > Only BOs this shrinker would actually back up have a say. BOs in > > > > XE_PL_SYSTEM, which are not GPU mapped and hence not part of a > > > > working > > > > set, and BOs xe_bo_shrink() rejects outright are never young. > > > > Purgeable > > > > BOs and those advised DONTNEED are neither: they are dropped > > > > rather > > > > than > > > > backed up and accounted to the purgeable shrinker, so they are > > > > never > > > > young and the count snoop skips their XE_BO_PRIORITY_LOW band > > > > entirely. > > > > > > > > The gate is hard and applies at every reclaim priority, so a > > > > working > > > > set > > > > kept younger than the delay is never backed up and can lead to > > > > OOM. > > > > This > > > > is deliberate, and why it is off by default. > > > > > > For completeness, Perhaps add to the commit message that > > > the plan is to follow up with an OOM notifier? > > > > > > > That is a possible option. Another idea I had was if we can get the > > priority exported from the shrinker core - we have a config knob that > > says 'at this priority or greater', shrink BOs which have been used > > recently. > > > > > I also wonder what happens during the PM notifier preparing for > > > suspend > > > on DGFX, Does that move mark bos accessed, hence possibly blocked > > > by > > > the shrinker at suspend time? > > > > > > > I haven't tried this on DGFX, can give it try but this off by > > default. > > > > I'm also a little confused by this - does the shrinker run at suspend > > time? I thought once the memory in ram a hiberation image in created > > from the that - i.e., there isn't a requirement the BOs to get shrunk > > into shmem. > > The problem occurs when evicting VRAM to system memory during the pm > notifier. If system fills up then the shrinker kicks in. But if that > eviction from VRAM to system doesn't bump LRU / update timestamps then > that is not a problem. > > For hibernation I'm not sure whether whether we actually see backup > shrinking, since it requires __GFP_FS and __GFP_IO for writeback, and I > think those are turned off during hibernation shrinking. > > > > > > > > > > > > Assisted-by: LLM > > > > Signed-off-by: Matthew Brost <[email protected]> > > > > --- > > > > drivers/gpu/drm/xe/Kconfig.profile | 15 +++ > > > > drivers/gpu/drm/xe/xe_bo.c | 83 ++++++++++++++ > > > > drivers/gpu/drm/xe/xe_bo.h | 5 + > > > > drivers/gpu/drm/xe/xe_bo_types.h | 8 ++ > > > > drivers/gpu/drm/xe/xe_device_sysfs.c | 35 ++++++ > > > > drivers/gpu/drm/xe/xe_pagefault.c | 2 +- > > > > drivers/gpu/drm/xe/xe_shrinker.c | 164 > > > > +++++++++++++++++++++++++-- > > > > drivers/gpu/drm/xe/xe_shrinker.h | 3 + > > > > drivers/gpu/drm/xe/xe_vm.c | 4 +- > > > > drivers/gpu/drm/xe/xe_vm_types.h | 8 ++ > > > > 10 files changed, 316 insertions(+), 11 deletions(-) > > > > > > > > diff --git a/drivers/gpu/drm/xe/Kconfig.profile > > > > b/drivers/gpu/drm/xe/Kconfig.profile > > > > index 23045ed2aa94..dd3985e502b8 100644 > > > > --- a/drivers/gpu/drm/xe/Kconfig.profile > > > > +++ b/drivers/gpu/drm/xe/Kconfig.profile > > > > @@ -74,6 +74,7 @@ config DRM_XE_ENABLE_SCHEDTIMEOUT_LIMIT > > > > to apply to applicable user. For elevated user, all > > > > above > > > > MIN > > > > and MAX values will apply when this configuration is > > > > enable to > > > > apply limitation. By default limitation is applied. > > > > + > > > > config DRM_XE_SHRINKER_NON_PURGEABLE_SEEKS > > > > int "Default seeks value of the non-purgeable bo > > > > shrinker" > > > > default 8 > > > > @@ -86,3 +87,17 @@ config DRM_XE_SHRINKER_NON_PURGEABLE_SEEKS > > > > reclaim pressure on non-purgeable buffer objects, > > > > which > > > > need > > > > GPU idling and writeback to be reclaimed, in favour of > > > > purgeable > > > > ones and other kernel caches. > > > > + > > > > +config DRM_XE_SHRINKER_MIN_AGE_MS > > > > + int "Default minimum BO age before shrinker backup (ms)" > > > > + default 0 > > > > + range 0 1000000 > > > > + help > > > > + Default for the per-device shrinker_min_age_ms sysfs > > > > attribute. The > > > > + shrinker won't back up a BO that was moved into TT, or > > > > bumped on the > > > > + LRU by an exec, less than this many milliseconds ago. > > > > BOs > > > > already > > > > + evicted to system memory are not part of a working set > > > > and > > > > are > > > > + never protected. This applies at every reclaim > > > > priority: a > > > > working > > > > + set kept younger than this is never backed up and can > > > > cause OOM. > > > > + Only set if that risk is preferred over thrashing a > > > > hot > > > > working > > > > + set. 0 disables. > > > > diff --git a/drivers/gpu/drm/xe/xe_bo.c > > > > b/drivers/gpu/drm/xe/xe_bo.c > > > > index 42a6aa1a2e4a..a4c0c6284a24 100644 > > > > --- a/drivers/gpu/drm/xe/xe_bo.c > > > > +++ b/drivers/gpu/drm/xe/xe_bo.c > > > > @@ -1357,9 +1357,92 @@ static int xe_bo_move(struct > > > > ttm_buffer_object > > > > *ttm_bo, bool evict, > > > > xe_tt_unmap_sg(xe, ttm_bo->ttm); > > > > } > > > > > > > > + /* > > > > + * ttm_resource_alloc() already placed the new resource > > > > at > > > > the tail of > > > > + * its LRU, or of @bo's bulk move, so only record when > > > > that > > > > happened. > > > > + */ > > > > + if (!ret && ttm_bo->resource && ttm_bo->resource- > > > > >mem_type > > > > == XE_PL_TT) > > > > + WRITE_ONCE(bo->lru_stamp, jiffies); > > > > + > > > > return ret; > > > > } > > > > > > > > +/** > > > > + * xe_bo_bump_lru_locked() - Bump a BO on the LRU and stamp its > > > > last > > > > use > > > > + * @bo: The buffer object, reserved by the caller. > > > > + * > > > > + * Move @bo to the tail of its LRU and record the time for the > > > > shrinker's > > > > + * minimum age gate. Only meaningful for BOs outside a bulk > > > > move, > > > > since a > > > > + * BO in one is anchored inside its range and is bumped, and > > > > aged, > > > > with it. > > > > + * The caller must hold the device LRU lock. > > > > + */ > > > > +void xe_bo_bump_lru_locked(struct xe_bo *bo) > > > > +{ > > > > + WRITE_ONCE(bo->lru_stamp, jiffies); > > > > + ttm_bo_move_to_lru_tail(&bo->ttm); > > > > +} > > > > + > > > > +/** > > > > + * xe_bo_shrink_backup_candidate() - Whether the shrinker may > > > > back > > > > up a BO > > > > + * @bo: The TTM buffer object, reserved by the caller. > > > > + * > > > > + * A BO is a backup candidate if backing it up is the only way > > > > to > > > > reclaim it. > > > > + * BOs advised DONTNEED, which sit at XE_BO_PRIORITY_LOW, and > > > > purgeable ones > > > > + * are not: their pages are simply dropped, which is cheap, and > > > > they > > > > are > > > > + * accounted to the purgeable shrinker. Neither are the BOs > > > > xe_bo_shrink() > > > > + * hands to xe_bo_shrink_purge() outright: ghost BOs, which > > > > aren't > > > > xe BOs, > > > > + * and BOs already freed and waiting to be reaped. Nor are BOs > > > > whose > > > > tt > > > > + * isn't TTM_TT_FLAG_EXTERNAL_MAPPABLE, which xe_bo_shrink() > > > > rejects. > > > > + * > > > > + * Return: true if @bo would be backed up by a non-purge shrink > > > > pass. > > > > + */ > > > > +bool xe_bo_shrink_backup_candidate(struct ttm_buffer_object *bo) > > > > +{ > > > > + struct xe_ttm_tt *xe_tt; > > > > + > > > > + if (!xe_bo_is_xe_bo(bo) || !kref_read(&bo- > > > > >base.refcount) || > > > > !bo->ttm || > > > > + !(bo->ttm->page_flags & > > > > TTM_TT_FLAG_EXTERNAL_MAPPABLE) > > > > > > > > > > + bo->priority <= XE_BO_PRIORITY_LOW) > > > > + return false; > > > > + > > > > + xe_tt = container_of(bo->ttm, struct xe_ttm_tt, ttm); > > > > + > > > > + return !xe_tt->purgeable; > > > > +} > > > > + > > > > +/** > > > > + * xe_bo_shrink_too_young() - Whether a BO was used too recently > > > > to > > > > shrink > > > > + * @bo: The TTM buffer object, reserved by the caller. > > > > + * @min_age: Minimum age in jiffies. 0 disables the check. > > > > + * > > > > + * The age is taken from whatever owns @bo's LRU position: > > > > &xe_vm.lru_stamp > > > > + * if @bo rides along with a VM's bulk move, its own > > > > &xe_bo.lru_stamp if not. > > > > + * > > > > + * BOs that the shrink walk would not back up anyway are never > > > > considered > > > > + * young, see xe_bo_shrink_backup_candidate(). Neither are BOs > > > > in > > > > + * XE_PL_SYSTEM: they are not GPU mapped, hence not part of a > > > > working set, > > > > + * and must be moved back to TT, refreshing the stamp, before > > > > use. > > > > + * > > > > + * Return: true if @bo was touched less than @min_age jiffies > > > > ago. > > > > + */ > > > > +bool xe_bo_shrink_too_young(struct ttm_buffer_object *bo, > > > > unsigned > > > > long min_age) > > > > +{ > > > > + struct xe_bo *xe_bo; > > > > + unsigned long stamp; > > > > + > > > > + if (!min_age || bo->resource->mem_type == XE_PL_SYSTEM > > > > || > > > > + !xe_bo_shrink_backup_candidate(bo)) > > > > + return false; > > > > + > > > > + xe_bo = ttm_to_xe_bo(bo); > > > > + if (bo->bulk_move && xe_bo->vm) > > > > + stamp = READ_ONCE(xe_bo->vm->lru_stamp); > > > > + else > > > > + stamp = READ_ONCE(xe_bo->lru_stamp); > > > > + > > > > + return stamp && time_before(jiffies, stamp + min_age); > > > > +} > > > > + > > > > static long xe_bo_shrink_purge(struct ttm_operation_ctx *ctx, > > > > struct ttm_buffer_object *bo, > > > > unsigned long *scanned) > > > > diff --git a/drivers/gpu/drm/xe/xe_bo.h > > > > b/drivers/gpu/drm/xe/xe_bo.h > > > > index 77b3a02e9b8e..65d4c638e8d8 100644 > > > > --- a/drivers/gpu/drm/xe/xe_bo.h > > > > +++ b/drivers/gpu/drm/xe/xe_bo.h > > > > @@ -623,6 +623,11 @@ struct xe_bo_shrink_flags { > > > > u32 writeback : 1; > > > > }; > > > > > > > > +void xe_bo_bump_lru_locked(struct xe_bo *bo); > > > > + > > > > +bool xe_bo_shrink_backup_candidate(struct ttm_buffer_object > > > > *bo); > > > > +bool xe_bo_shrink_too_young(struct ttm_buffer_object *bo, > > > > unsigned > > > > long min_age); > > > > + > > > > long xe_bo_shrink(struct ttm_operation_ctx *ctx, struct > > > > ttm_buffer_object *bo, > > > > const struct xe_bo_shrink_flags flags, > > > > unsigned long *scanned); > > > > diff --git a/drivers/gpu/drm/xe/xe_bo_types.h > > > > b/drivers/gpu/drm/xe/xe_bo_types.h > > > > index 0265fb314c77..f63b668c7134 100644 > > > > --- a/drivers/gpu/drm/xe/xe_bo_types.h > > > > +++ b/drivers/gpu/drm/xe/xe_bo_types.h > > > > @@ -49,6 +49,14 @@ struct xe_bo { > > > > u32 flags; > > > > /** @vm: VM this BO is attached to, for extobj this will > > > > be > > > > NULL */ > > > > struct xe_vm *vm; > > > > + /** > > > > + * @lru_stamp: jiffies at the last LRU bump, done on a > > > > move > > > > into TT > > > > + * and at exec time. Only describes the LRU position of > > > > a BO > > > > outside > > > > + * a bulk move; BOs in one are aged by &xe_vm.lru_stamp > > > > instead. > > > > + * Consulted by the shrinker's minimum age gate. > > > > Accessed > > > > with > > > > + * READ_ONCE()/WRITE_ONCE(). > > > > + */ > > > > + unsigned long lru_stamp; > > > > /** > > > > * @q: Queue this BO is attached to, mostly for LRC BO, > > > > NULL > > > > otherwise. > > > > * Protected by the BO dma_resv: readers must hold it > > > > across > > > > both the > > > > diff --git a/drivers/gpu/drm/xe/xe_device_sysfs.c > > > > b/drivers/gpu/drm/xe/xe_device_sysfs.c > > > > index ff5d25dd867c..26b1f71ed522 100644 > > > > --- a/drivers/gpu/drm/xe/xe_device_sysfs.c > > > > +++ b/drivers/gpu/drm/xe/xe_device_sysfs.c > > > > @@ -38,6 +38,12 @@ > > > > * reclaimed by the shrinker. Higher values reduce reclaim > > > > pressure > > > > on > > > > * non-purgeable buffer objects. Valid values are 1 to 64. > > > > Defaults > > > > to > > > > * CONFIG_DRM_XE_SHRINKER_NON_PURGEABLE_SEEKS. > > > > + * > > > > + * shrinker_min_age_ms - Report/change the minimum time, in ms, > > > > since a BO was > > > > + * last moved to TT or bumped on the LRU by an exec before the > > > > shrinker may > > > > + * back it up. BOs evicted to system memory are never protected. > > > > Defaults to > > > > + * CONFIG_DRM_XE_SHRINKER_MIN_AGE_MS, 0 disables. This applies > > > > at > > > > every > > > > + * reclaim priority, so a non-zero value can lead to OOM. > > > > */ > > > > > > > > static ssize_t > > > > @@ -331,8 +337,37 @@ shrinker_non_purgeable_seeks_store(struct > > > > device > > > > *dev, > > > > > > > > static DEVICE_ATTR_RW(shrinker_non_purgeable_seeks); > > > > > > > > +static ssize_t > > > > +shrinker_min_age_ms_show(struct device *dev, struct > > > > device_attribute > > > > *attr, > > > > + char *buf) > > > > +{ > > > > + struct xe_device *xe = > > > > pdev_to_xe_device(to_pci_dev(dev)); > > > > + > > > > + return sysfs_emit(buf, "%u\n", > > > > xe_shrinker_min_age_ms(xe- > > > > > mem.shrinker)); > > > > +} > > > > + > > > > +static ssize_t > > > > +shrinker_min_age_ms_store(struct device *dev, struct > > > > device_attribute *attr, > > > > + const char *buff, size_t count) > > > > +{ > > > > + struct xe_device *xe = > > > > pdev_to_xe_device(to_pci_dev(dev)); > > > > + unsigned int ms; > > > > + int ret; > > > > + > > > > + ret = kstrtouint(buff, 0, &ms); > > > > + if (ret) > > > > + return ret; > > > > + > > > > + xe_shrinker_set_min_age_ms(xe->mem.shrinker, ms); > > > > + > > > > + return count; > > > > +} > > > > + > > > > +static DEVICE_ATTR_RW(shrinker_min_age_ms); > > > > + > > > > static struct attribute *shrinker_attrs[] = { > > > > &dev_attr_shrinker_non_purgeable_seeks.attr, > > > > + &dev_attr_shrinker_min_age_ms.attr, > > > > NULL > > > > }; > > > > > > > > diff --git a/drivers/gpu/drm/xe/xe_pagefault.c > > > > b/drivers/gpu/drm/xe/xe_pagefault.c > > > > index 6828f81df50e..895866d639f8 100644 > > > > --- a/drivers/gpu/drm/xe/xe_pagefault.c > > > > +++ b/drivers/gpu/drm/xe/xe_pagefault.c > > > > @@ -226,7 +226,7 @@ static int xe_pagefault_handle_vma(struct > > > > xe_gt > > > > *gt, struct xe_vma *vma, > > > > > > > > if (bo) { > > > > scoped_guard(spinlock, &vm->xe- > > > > > ttm.lru_lock) > > > > - > > > > ttm_bo_move_to_lru_tail(&bo- > > > > > ttm); > > > > + xe_bo_bump_lru_locked(bo > > > > ); > > > > } > > > > } > > > > xe_validation_ctx_fini(&ctx); > > > > diff --git a/drivers/gpu/drm/xe/xe_shrinker.c > > > > b/drivers/gpu/drm/xe/xe_shrinker.c > > > > index c86455ea08d1..64e5e684f591 100644 > > > > --- a/drivers/gpu/drm/xe/xe_shrinker.c > > > > +++ b/drivers/gpu/drm/xe/xe_shrinker.c > > > > @@ -39,6 +39,8 @@ > > > > * @shrink: Pointer to the mm shrinker handling non-purgeable > > > > (shrinkable) objects. > > > > * @purge_shrink: Pointer to the mm shrinker handling purgeable > > > > objects. > > > > * @pm_worker: Worker to wake up the device if required. > > > > + * @min_age_ms: Minimum BO age before backup, in ms. 0 disables. > > > > Defaults to > > > > + * CONFIG_DRM_XE_SHRINKER_MIN_AGE_MS, overridable through sysfs. > > > > */ > > > > struct xe_shrinker { > > > > struct xe_device *xe; > > > > @@ -50,6 +52,7 @@ struct xe_shrinker { > > > > struct shrinker *shrink; > > > > struct shrinker *purge_shrink; > > > > struct work_struct pm_worker; > > > > + unsigned int min_age_ms; > > > > }; > > > > > > > > static struct xe_shrinker *to_xe_shrinker(struct shrinker > > > > *shrink) > > > > @@ -104,6 +107,7 @@ static void xe_shrinker_runtime_pm_put(struct > > > > xe_shrinker *shrinker, bool runtim > > > > static int __xe_shrinker_walk(struct xe_shrinker *shrinker, > > > > struct ttm_operation_ctx *ctx, > > > > const struct xe_bo_shrink_flags > > > > flags, > > > > + unsigned long min_age, > > > > unsigned long to_scan, unsigned > > > > long > > > > *scanned, > > > > unsigned long *freed) > > > > { > > > > @@ -136,6 +140,16 @@ static int __xe_shrinker_walk(struct > > > > xe_shrinker > > > > *shrinker, > > > > if (flags.purge && ttm_bo->priority > > > > > XE_BO_PRIORITY_LOW) > > > > break; > > > > > > > > + /* > > > > + * Each LRU priority is roughly age > > > > ordered, > > > > so the > > > > + * rest of this priority is most likely > > > > young too. > > > > + * Move on to the next priority. > > > > + */ > > > > + if (xe_bo_shrink_too_young(ttm_bo, > > > > min_age)) > > > > { > > > > + ttm_bo_lru_cursor_skip_priority( > > > > &cur > > > > s); > > > > + continue; > > > > + } > > > > + > > > > if (!ttm_bo_shrink_suitable(ttm_bo, > > > > ctx)) > > > > continue; > > > > > > > > @@ -171,6 +185,7 @@ static int __xe_shrinker_walk(struct > > > > xe_shrinker > > > > *shrinker, > > > > static int xe_shrinker_walk(struct xe_shrinker *shrinker, > > > > struct ttm_operation_ctx *ctx, > > > > const struct xe_bo_shrink_flags > > > > flags, > > > > + unsigned long min_age, > > > > unsigned long to_scan, unsigned long > > > > *scanned, > > > > unsigned long *freed) > > > > { > > > > @@ -180,26 +195,126 @@ static int xe_shrinker_walk(struct > > > > xe_shrinker > > > > *shrinker, > > > > > > > > swap(no_wait_gpu, ctx->no_wait_gpu); > > > > save_flags.writeback = false; > > > > - ret = __xe_shrinker_walk(shrinker, ctx, save_flags, > > > > to_scan, > > > > scanned, > > > > - freed); > > > > + ret = __xe_shrinker_walk(shrinker, ctx, save_flags, > > > > min_age, > > > > + to_scan, scanned, freed); > > > > swap(no_wait_gpu, ctx->no_wait_gpu); > > > > if (ret || *scanned >= to_scan) > > > > return ret; > > > > > > > > if (!ctx->no_wait_gpu) { > > > > - ret = __xe_shrinker_walk(shrinker, ctx, > > > > save_flags, > > > > to_scan, scanned, > > > > - freed); > > > > + ret = __xe_shrinker_walk(shrinker, ctx, > > > > save_flags, > > > > min_age, > > > > + to_scan, scanned, > > > > freed); > > > > if (ret || *scanned >= to_scan) > > > > return ret; > > > > } > > > > > > > > if (flags.writeback) > > > > - ret = __xe_shrinker_walk(shrinker, ctx, flags, > > > > to_scan, scanned, > > > > - freed); > > > > + ret = __xe_shrinker_walk(shrinker, ctx, flags, > > > > min_age, > > > > + to_scan, scanned, > > > > freed); > > > > > > > > return ret; > > > > } > > > > > > > > +/** > > > > + * DOC: Shrinker minimum age > > > > + * > > > > + * The per-device shrinker_min_age_ms sysfs attribute, > > > > defaulting to > > > > + * CONFIG_DRM_XE_SHRINKER_MIN_AGE_MS, protects recently used BOs > > > > from backup. > > > > + * A BO's age is taken from whatever owns its position on the > > > > LRU, > > > > and is > > > > + * stamped wherever that position is updated: the VM-wide stamp > > > > for > > > > a BO > > > > + * riding along with a VM's bulk move, bumped on exec in > > > > xe_vm_bump_lru(), > > > > + * its own stamp otherwise, bumped there as well and on a move > > > > into > > > > TT in > > > > + * xe_bo_move(). BOs in XE_PL_SYSTEM are never young: they are > > > > not > > > > GPU > > > > + * mapped, so not part of a working set, and will be stamped > > > > again > > > > when > > > > + * moved back to TT for use. > > > > + * > > > > + * The gate is applied per LRU priority, each of which is > > > > roughly > > > > age > > > > + * ordered. xe_shrinker_count() snoops the first shrinkable BO > > > > of > > > > each > > > > + * priority on each TT-backed LRU. Only if all of them are young > > > > does it stop > > > > + * reporting shrinkable pages; anything else, including reaching > > > > no > > > > verdict > > > > + * at all, lets it report. The backup walk in xe_shrinker_scan() > > > > skips to the > > > > + * next priority at the first young BO, as a backstop since the > > > > count snoop > > > > + * is only a sample. A hot priority thus doesn't shield colder > > > > BOs > > > > in another > > > > + * priority. BOs that aren't backed up anyway have no say either > > > > way: those > > > > + * in XE_PL_SYSTEM are never young, and purgeable and DONTNEED > > > > ones, > > > > which > > > > + * belong to the purgeable shrinker, are ignored by the snoop > > > > entirely. > > > > + * > > > > + * The gate is hard and applies at every reclaim priority. A > > > > working > > > > set kept > > > > + * younger than the delay is never backed up, which can lead to > > > > OOM. > > > > This is > > > > + * intended for users who prefer that risk over thrashing a hot > > > > working set, > > > > + * hence it is disabled by default. > > > > + */ > > > > +static unsigned long xe_shrinker_min_age(struct xe_shrinker > > > > *shrinker) > > > > +{ > > > > + unsigned int ms = READ_ONCE(shrinker->min_age_ms); > > > > + > > > > + return ms ? msecs_to_jiffies(ms) : 0; > > > > +} > > > > + > > > > +/* Max shrink-unsuitable BOs to skip per LRU before giving up > > > > the > > > > snoop. */ > > > > +#define XE_SHRINKER_SNOOP_MAX 16 > > > > + > > > > +/* > > > > + * Whether the first BO of every priority on every TT-backed LRU > > > > that this > > > > + * shrinker would back up is younger than @min_age. BOs are > > > > trylocked through > > > > + * the reserved LRU walker since their tt may be torn down while > > > > still on the > > > > + * LRU. Contended BOs are skipped by the walker, and may be all > > > > of > > > > them. If no > > > > + * verdict can be reached, because nothing was inspected or the > > > > snoop budget > > > > + * ran out, report not young and leave it to the scan side > > > > check. > > > > + */ > > > > +static bool xe_shrinker_lru_young(struct xe_shrinker *shrinker, > > > > + unsigned long min_age) > > > > +{ > > > > + struct ttm_operation_ctx ctx = { > > > > + .interruptible = false, > > > > + .no_wait_gpu = false, > > > > + }; > > > > + struct xe_device *xe = shrinker->xe; > > > > + unsigned int mem_type; > > > > + bool young = false; > > > > + > > > > + for (mem_type = XE_PL_SYSTEM; mem_type <= XE_PL_TT; > > > > ++mem_type) { > > > > + struct ttm_resource_manager *man = > > > > ttm_manager_type(&xe->ttm, mem_type); > > > > + struct ttm_bo_lru_cursor curs; > > > > + struct ttm_buffer_object *ttm_bo; > > > > + struct ttm_lru_walk_arg arg = { > > > > + .ctx = &ctx, > > > > + .trylock_only = true, > > > > + }; > > > > + unsigned int budget = XE_SHRINKER_SNOOP_MAX; > > > > + > > > > + if (!man || !man->use_tt) > > > > + continue; > > > > + > > > > + ttm_bo_lru_for_each_reserved_guarded(&curs, man, > > > > &arg, ttm_bo) { > > > > + /* > > > > + * Nothing in the DONTNEED band is > > > > backed > > > > up, and it > > > > + * is accounted to the purgeable > > > > shrinker, > > > > so it has > > > > + * no say here. Skip the whole band in > > > > one > > > > go. > > > > + */ > > > > + if (ttm_bo->priority <= > > > > XE_BO_PRIORITY_LOW) > > > > { > > > > + ttm_bo_lru_cursor_skip_priority( > > > > &cur > > > > s); > > > > + continue; > > > > + } > > > > + > > > > + if (!ttm_bo_shrink_suitable(ttm_bo, > > > > &ctx) || > > > > + > > > > !xe_bo_shrink_backup_candidate(ttm_bo)) > > > > { > > > > + if (!--budget) > > > > + return false; > > > > + continue; > > > > + } > > > > + > > > > + if (!xe_bo_shrink_too_young(ttm_bo, > > > > min_age)) > > > > + return false; > > > > + > > > > + young = true; > > > > + ttm_bo_lru_cursor_skip_priority(&curs); > > > > + } > > > > + } > > > > + > > > > + return young; > > > > +} > > > > + > > > > static unsigned long > > > > xe_shrinker_count_purgeable(struct shrinker *shrink, struct > > > > shrink_control *sc) > > > > { > > > > @@ -239,7 +354,7 @@ static unsigned long > > > > xe_shrinker_count(struct shrinker *shrink, struct shrink_control > > > > *sc) > > > > { > > > > struct xe_shrinker *shrinker = to_xe_shrinker(shrink); > > > > - unsigned long num_pages; > > > > + unsigned long num_pages, min_age; > > > > bool can_backup = !!(sc->gfp_mask & __GFP_FS); > > > > > > > > if (!can_backup) > > > > @@ -263,6 +378,10 @@ xe_shrinker_count(struct shrinker *shrink, > > > > struct shrink_control *sc) > > > > > > > > write_unlock(&shrinker->lock); > > > > > > > > + min_age = xe_shrinker_min_age(shrinker); > > > > + if (num_pages && min_age && > > > > xe_shrinker_lru_young(shrinker, > > > > min_age)) > > > > + return SHRINK_EMPTY; > > > > + > > > > return num_pages ? num_pages : SHRINK_EMPTY; > > > > } > > > > > > > > @@ -297,7 +416,8 @@ static unsigned long > > > > xe_shrinker_scan_purgeable(struct shrinker *shrink, > > > > .purge = true, > > > > }; > > > > > > > > - xe_shrinker_walk(shrinker, &ctx, shrink_flags, > > > > nr_to_scan, > > > > &nr_scanned, &freed); > > > > + xe_shrinker_walk(shrinker, &ctx, shrink_flags, 0, > > > > nr_to_scan, &nr_scanned, > > > > + &freed); > > > > > > > > sc->nr_scanned = nr_scanned; > > > > > > > > @@ -322,7 +442,9 @@ static unsigned long xe_shrinker_scan(struct > > > > shrinker *shrink, struct shrink_con > > > > /* Might need runtime PM to access backed-up or CCS- > > > > compressed data. */ > > > > runtime_pm = xe_shrinker_runtime_pm_get(shrinker); > > > > > > > > - xe_shrinker_walk(shrinker, &ctx, shrink_flags, > > > > nr_to_scan, > > > > &nr_scanned, &freed); > > > > + xe_shrinker_walk(shrinker, &ctx, shrink_flags, > > > > + xe_shrinker_min_age(shrinker), > > > > nr_to_scan, > > > > &nr_scanned, > > > > + &freed); > > > > > > > > sc->nr_scanned = nr_scanned; > > > > xe_shrinker_runtime_pm_put(shrinker, runtime_pm); > > > > @@ -387,6 +509,29 @@ int > > > > xe_shrinker_set_non_purgeable_seeks(struct > > > > xe_shrinker *shrinker, > > > > return 0; > > > > } > > > > > > > > +/** > > > > + * xe_shrinker_min_age_ms() - Get the shrinker minimum BO age > > > > + * @shrinker: Pointer to the struct xe_shrinker. > > > > + * > > > > + * Return: Minimum age, in ms, a BO must reach before it may be > > > > backed up. > > > > + */ > > > > +unsigned int xe_shrinker_min_age_ms(struct xe_shrinker > > > > *shrinker) > > > > +{ > > > > + return READ_ONCE(shrinker->min_age_ms); > > > > +} > > > > + > > > > +/** > > > > + * xe_shrinker_set_min_age_ms() - Set the shrinker minimum BO > > > > age > > > > + * @shrinker: Pointer to the struct xe_shrinker. > > > > + * @ms: Minimum age, in ms. 0 disables the gate. > > > > + * > > > > + * See DOC: Shrinker minimum age. A non-zero value can lead to > > > > OOM. > > > > + */ > > > > +void xe_shrinker_set_min_age_ms(struct xe_shrinker *shrinker, > > > > unsigned int ms) > > > > +{ > > > > + WRITE_ONCE(shrinker->min_age_ms, ms); > > > > +} > > > > + > > > > /** > > > > * xe_shrinker_create() - Create an xe per-device shrinker > > > > * @xe: Pointer to the xe device. > > > > @@ -415,6 +560,7 @@ int xe_shrinker_create(struct xe_device *xe) > > > > > > > > INIT_WORK(&shrinker->pm_worker, xe_shrinker_pm); > > > > shrinker->xe = xe; > > > > + shrinker->min_age_ms = > > > > CONFIG_DRM_XE_SHRINKER_MIN_AGE_MS; > > > > rwlock_init(&shrinker->lock); > > > > > > > > shrinker->purge_shrink->count_objects = > > > > xe_shrinker_count_purgeable; > > > > diff --git a/drivers/gpu/drm/xe/xe_shrinker.h > > > > b/drivers/gpu/drm/xe/xe_shrinker.h > > > > index 4b3c73689619..586d7772bd58 100644 > > > > --- a/drivers/gpu/drm/xe/xe_shrinker.h > > > > +++ b/drivers/gpu/drm/xe/xe_shrinker.h > > > > @@ -18,4 +18,7 @@ int xe_shrinker_non_purgeable_seeks(struct > > > > xe_shrinker *shrinker); > > > > int xe_shrinker_set_non_purgeable_seeks(struct xe_shrinker > > > > *shrinker, > > > > unsigned int seeks); > > > > > > > > +unsigned int xe_shrinker_min_age_ms(struct xe_shrinker > > > > *shrinker); > > > > +void xe_shrinker_set_min_age_ms(struct xe_shrinker *shrinker, > > > > unsigned int ms); > > > > + > > > > #endif > > > > diff --git a/drivers/gpu/drm/xe/xe_vm.c > > > > b/drivers/gpu/drm/xe/xe_vm.c > > > > index 86c3b4fd232f..583c55b7f97d 100644 > > > > --- a/drivers/gpu/drm/xe/xe_vm.c > > > > +++ b/drivers/gpu/drm/xe/xe_vm.c > > > > @@ -463,13 +463,15 @@ void xe_vm_bump_lru(struct xe_vm *vm, > > > > struct > > > > drm_exec *exec) > > > > spinlock_t *lru_lock = &vm->xe->ttm.lru_lock; > > > > struct drm_gem_object *obj; > > > > > > > > + WRITE_ONCE(vm->lru_stamp, jiffies); > > > > + > > > > spin_lock(lru_lock); > > > > ttm_lru_bulk_move_tail(&vm->lru_bulk_move); > > > > drm_exec_for_each_locked_object(exec, obj) { > > > > if (!drm_gpuvm_is_extobj(&vm->gpuvm, obj)) > > > > continue; > > > > > > > > - ttm_bo_move_to_lru_tail(&gem_to_xe_bo(obj)- > > > > >ttm); > > > > + xe_bo_bump_lru_locked(gem_to_xe_bo(obj)); > > > > cond_resched_lock(lru_lock); > > > > } > > > > spin_unlock(lru_lock); > > > > diff --git a/drivers/gpu/drm/xe/xe_vm_types.h > > > > b/drivers/gpu/drm/xe/xe_vm_types.h > > > > index fb9305124679..046505a285e9 100644 > > > > --- a/drivers/gpu/drm/xe/xe_vm_types.h > > > > +++ b/drivers/gpu/drm/xe/xe_vm_types.h > > > > @@ -267,6 +267,14 @@ struct xe_vm { > > > > /** @lru_bulk_move: Bulk LRU move list for this VM's BOs > > > > */ > > > > struct ttm_lru_bulk_move lru_bulk_move; > > > > > > > > + /** > > > > + * @lru_stamp: jiffies at the last xe_vm_bump_lru(), > > > > which > > > > is where > > > > + * @lru_bulk_move is moved to the LRU tail. Ages every > > > > BO in > > > > that > > > > + * bulk move for the shrinker's minimum age gate. > > > > Accessed > > > > with > > > > + * READ_ONCE()/WRITE_ONCE(). > > > > + */ > > > > + unsigned long lru_stamp; > > > > > > Isn't the bo resv / vm resv always held then this is accessed? > > > > > > > xe_bo_shrink_too_young is the lockless read side. > > Hm. To me it looks like all accesses have the bo resv held, eliminating > the need for READ_ONCE() / WRITE_ONCE(), and xe_bo_shrink_too_young() > also requires resv? >
I think you are right, let me drop the READ_ONCE/WRITE_ONCE + add lock annotations to xe_bo_shrink_too_young. > Also another thing that came up is that during a pagefault, for a vm > bo, the code timestamps the bo (which IMO is correct), but the shrinker > reads the vm timestamp and never picks up the bo timestamp. I believe > the shrinker should pick the younger of the two. > In the case of page faults bo->bulk_move will be NULL so BO's timestamp is always read. Matt > Thanks, > Thomas > > > > > > Matt > > > > > Thanks, > > > Thomas > > > > > > > > > > > > > + > > > > u64 size; > > > > > > > > struct xe_pt *pt_root[XE_MAX_TILES_PER_DEVICE];
