On Mon, Mar 10, 2025 at 01:06:06PM +0100, Maxime Ripard wrote: > Hi, > > Here's preliminary work to enable dmem tracking for heavy users of DMA > allocations on behalf of userspace: v4l2, DRM, and dma-buf heaps. > > It's not really meant for inclusion at the moment, because I really > don't like it that much, and would like to discuss solutions on how to > make it nicer. > > In particular, the dma dmem region accessors don't feel that great to > me. It duplicates the logic to select the proper accessor in > dma_alloc_attrs(), and it looks fragile and potentially buggy to me. > > One solution I tried is to do the accounting in dma_alloc_attrs() > directly, depending on a flag being set, similar to what __GFP_ACCOUNT > is doing. > > It didn't work because dmem initialises a state pointer when charging an > allocation to a region, and expects that state pointer to be passed back > when uncharging. Since dma_alloc_attrs() returns a void pointer to the > allocated buffer, we need to put that state into a higher-level > structure, such as drm_gem_object, or dma_buf. > > Since we can't share the region selection logic, we need to get the > region through some other mean. Another thing I consider was to return > the region as part of the allocated buffer (through struct page or > folio), but those are lost across the calls and dma_alloc_attrs() will > only get a void pointer. So that's not doable without some heavy > rework, if it's a good idea at all. > > So yeah, I went for the dumbest possible solution with the accessors, > hoping you could suggest a much smarter idea :)
I've had a private chat with Maxime to get him up to speed on hopefully a lot of the past discussions, but probably best I put my notes here too. Somewhat unstructured list of challenges with trying to account all the memory for gpu/isp/camera/whatever: - At LPC in Dublin I think we've pretty much reached the conclusion that normal struct page memory should be just accounted in memcg. Otherwise you just get really nasty double-accounting chaos or issues where you can exhaust reserves. - We did not figure out what to do with mixed stuff like CMA, where we probably want to account it both into memcg (because it's struct page) but also separately into dmem (because the CMA region is a limited resource and only using memcg will not help us manage it). - There's the entire chaos of carve-out vs CMA and how userspace can figure out how to set reasonable limits automatically. Maxime brought the issue that limits need to be adjusted if carve-out/CMA/shmem aren't accounted the same, which I think is a valid concern. But due to the above conclusion around memcg accounting I think that's unavoidable, so we need some means for userspace to autoconfigure reasonable limits. Then that autoconfig can be done on each boot, and kernel (or dt or whatever) changes between these three allocators don't matter anymore. - Autoconfiguration challenges also exist for split display/render SoC. It gets even more fun if you also throw in camera and media codecs, and even more fun if you have multiple CMA regions. - Discrete gpu also has a very fun autoconfiguration issue because you have dmem limits for vram, and memcg limits for system memory. Vram might be swapped out to system memory, so naively you might want to assume that you need higher memcg limits than dmem limits. But there's systems with more dmem and smem (because the cpu with its memory is essentially just the co-processor that orchestrates the real compute machine, which is all gpus). - We need a charge transfer, least for Android since there all memory is allocated through binder. TJ Mercier did some patches: https://lore.kernel.org/dri-devel/20230123191728.2928839-3-tjmerc...@google.com/ Ofc with dmem this would need to work for both dmem and memcg charges, since with CMA and discrete gpu we'll have bo that are tracked in both. - Hard limits for shmem/ttm drivers need a memcg-aware shrinker. TTM doesn't even have a shrinker yet, but with xe we now have a helper-library approach to enabling shrinking for TTM drivers. memcg-aware shrinking will be a large step up in complexity on top (and probably a good reason to switch over to the common shrinker lru instead of hand-rolling). See the various attempts at ttm shrinkers by Christian König and Thomas Hellstrom over the past years on dri-devel. This also means that most likely cgroup limit enforcement for ttm based drivers will be per-driver or at least very uneven. - Hard limits for dmem vram means ttm eviction needs to be able to account the evicted bo against the right memcg. Because this can happen in random other threads (cs ioctl of another process, kernel threads) accounting this correctly is going to be "fun". Plus I haven't thought through interactions with memcg-aware shrinkers, which might cause some really fundamental issues. - We also ideally need pin account, but I don't think we have any consensus on how to do that for memcg memory. Thus far it's all functionality-specific limits (e.g. mlock, rdma has its own for long-term pinned memory), not sure it makes sense to push for a unified tracking in memcg here? For dmem I think it's pretty easy, but there the question is how to differentiate between dmem that's always pinned (cma, I don't think anyone bothered with a shrinker for cma memory, vc4 maybe?) and dmem that generally has a shrinker and really wants a separate pin limit (vram/ttm drivers). - Unfortunately on top of the sometimes very high individual complexity these issues also all interact. Which means that we won't be able to roll this out in one go, and we need to cope with very uneven enforcement. I think trying to allow userspace to cope with changing cgroup support through autoconfiguration is the most feasible way out of this challenge. tldr; cgroup for device memory is a really complex mess Cheers, Sima > Thanks, > Maxime > > Signed-off-by: Maxime Ripard <mrip...@kernel.org> > --- > Maxime Ripard (12): > cma: Register dmem region for each cma region > cma: Provide accessor to cma dmem region > dma: coherent: Register dmem region for each coherent region > dma: coherent: Provide accessor to dmem region > dma: contiguous: Provide accessor to dmem region > dma: direct: Provide accessor to dmem region > dma: Create default dmem region for DMA allocations > dma: Provide accessor to dmem region > dma-buf: Clear cgroup accounting on release > dma-buf: cma: Account for allocations in dmem cgroup > drm/gem: Add cgroup memory accounting > media: videobuf2: Track buffer allocations through the dmem cgroup > > drivers/dma-buf/dma-buf.c | 7 ++++ > drivers/dma-buf/heaps/cma_heap.c | 18 ++++++++-- > drivers/gpu/drm/drm_gem.c | 5 +++ > drivers/gpu/drm/drm_gem_dma_helper.c | 6 ++++ > .../media/common/videobuf2/videobuf2-dma-contig.c | 19 +++++++++++ > include/drm/drm_device.h | 1 + > include/drm/drm_gem.h | 2 ++ > include/linux/cma.h | 9 +++++ > include/linux/dma-buf.h | 5 +++ > include/linux/dma-direct.h | 2 ++ > include/linux/dma-map-ops.h | 32 ++++++++++++++++++ > include/linux/dma-mapping.h | 11 ++++++ > kernel/dma/coherent.c | 26 +++++++++++++++ > kernel/dma/direct.c | 8 +++++ > kernel/dma/mapping.c | 39 > ++++++++++++++++++++++ > mm/cma.c | 21 +++++++++++- > mm/cma.h | 3 ++ > 17 files changed, 211 insertions(+), 3 deletions(-) > --- > base-commit: 55a2aa61ba59c138bd956afe0376ec412a7004cf > change-id: 20250307-dmem-cgroups-73febced0989 > > Best regards, > -- > Maxime Ripard <mrip...@kernel.org> > -- Simona Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
