> The long-term solution That was the part that I missed in the description. Please note somewhere that we still need to improve this.
Apart from that the patches look fine to me, but I need more time to review them in detail. Regards, Christian. Am 31.12.2014 um 15:06 schrieb Oded Gabbay: > > On 12/31/2014 03:49 PM, Christian König wrote: >> Am 31.12.2014 um 14:39 schrieb Oded Gabbay: >>> Background: >>> >>> amdkfd needs GART memory for several things, such as runlist packets, >>> MQDs, HPDs and more. Unfortunately, all of this memory must be always >>> pinned (due to several reasons which were discussed during the >>> initial review of amdkfd). >> In general seems to be a good idea, but so far I still don't have seen a >> good explanation why all those memory must be pinned. So please summarize >> that one once more. >> >> Regards, >> Christian. >> > ok, once more :) > > The bulk of the allocations in the GART is for MQDs. MQDs represent active > user-mode queues, which are on the current runlist. It is important to > remember that active queues doesn't necessarily mean scheduled/running > queues, especially if there is over-subscription of queues or more than a > single HSA process. > > Because the scheduling of the user-mode queues is done by the CP firmware, > amdkfd doesn't have any indication if the queue is scheduled or not. If the > CP will try to schedule a queue, and its MQD is not present, this will > probably stuck the CP permanently, as it will load garbage from the GART > (the address of the MQD is given to the CP inside the runlist packet). > > In addition, there are a couple of small allocations which also should > always be pinned - runlist packets (2 packets) and HPDs. runlist packets can > be quite large, depending on number of processes and queues. > > A few solutions were proposed, but at the end Jerome agreed there is no harm > when limiting the total memory consumption to around 4MB. > > The long-term solution, which I will be working on, hopefully soon, is to > create a mechanism through which radeon/ttm can ask amdkfd to clear > GART/VRAM memory due to memory pressure. Then, amdkfd will preempt the > running queues and wait until the memory pressure is over. Then it will > reschedule the queues. But I'm getting ahead of myself. I hope to send an > RFC about that in the next couple of weeks. > > Oded > > > >>> Current Solution: >>> >>> The current (short/mid-term) solution that was proposed by Jerome.G, is >>> to limit the amount of memory to a small size, roughly 4MB and allocate >>> this buffer at the start of the GART. To accomodate this, amdkfd has >>> two kernel module parameters, maximum number of HSA processes and >>> maximum number of queues per process, which require under 4MB of GART >>> memory when using their defaults, 32 and 128 respectively. >>> >>> Until now, amdkfd used the radeon sub-allocator module (radeon_sa) >>> to handle the sub-allocation of memory from this large buffer to >>> different modules inside the amdkfd. >>> >>> However, while running OpenCL conformance test suite, we found that >>> radeon_sa module is not suitable for this kind of task, due to its >>> design: >>> 1. Every allocation increments its interal pointer so the next >>> allocation is *always* done ahead of the previous allocation. This >>> causes the internal pointer to wrap-around when it reaches the end of >>> the buffer. >>> >>> 2. When encoutering an area that is already allocated, the module >>> waits for that area to be freed. If it is not freed in a timely manner >>> (or has no fence), the allocation fails. Simply put, it can't "skip" >>> the allocated area. >>> >>> Now, this is most probably good for graphics, but for amdkfd needs, >>> the combination of the two behaviors mentioned above eventually causes >>> a denial-of-service. This is because some memory allocations >>> are *always* present and *never* freed (such as HPDs). >>> Therefore, given enough time and workload, the radeon_sa eventually >>> wraps around, encounters an already allocated area and gets stuck. >>> >>> Proposed new solution: >>> >>> To solve this, I have written a simple sub-allocator module inside >>> amdkfd. It allocates fixed-size contiguous chunks (1 or more) and uses >>> a bitmap to manage the allocations. The next allocation is always >>> being searched for from the start of the GART buffer, and the module >>> knows how to skip allocated chunks. >>> >>> Because most allocations are MQDs, and MQDs are 512 Bytes in size, I >>> set the default chunk size to be 512 Bytes. >>> >>> The basic GART memory allocation is still being done in the >>> amdkfd <--> radeon interface, and it still occupies less than 4MB. >>> >>> I have chosen to implement a new allocator instead of changing >>> radeon_sa because the behavior of radeon_sa is very appropriate for >>> graphics, where allocations do not stay forever. Also, amdkfd doesn't >>> actually need the flexibility and features radeon_sa provides. >>> >>> Oded >>> >>> Oded Gabbay (9): >>> drm/amd: Add new kfd-->kgd interface for gart usage >>> drm/radeon: Impl. new gtt allocate/free functions >>> drm/amdkfd: Add gtt sa related data to kfd_dev struct >>> drm/amdkfd: Add kfd gtt sub-allocator functions >>> drm/amdkfd: Fixed calculation of gart buffer size >>> drm/amdkfd: Allocate gart memory using new interface >>> drm/amdkfd: Using new gtt sa in amdkfd >>> drm/radeon: Remove old radeon_sa usage from kfd-->kgd interface >>> drm/amd: Remove old radeon_sa funcs from kfd-->kgd interface >>> >>> drivers/gpu/drm/amd/amdkfd/kfd_device.c | 217 >>> ++++++++++++++++++++- >>> .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.c | 23 +-- >>> drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.c | 41 ++-- >>> drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager.c | 16 +- >>> drivers/gpu/drm/amd/amdkfd/kfd_packet_manager.c | 10 +- >>> drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 28 ++- >>> drivers/gpu/drm/amd/include/kgd_kfd_interface.h | 23 +-- >>> drivers/gpu/drm/radeon/radeon_kfd.c | 128 ++++++------ >>> 8 files changed, 329 insertions(+), 157 deletions(-) >>>
