Re: [PATCH] drm/amdkfd: Remove Align VRAM allocations to 1MB on APU ASIC

[Kuehling, Felix]

[AMD Official Use Only - General]

As a compromise we are considering a change that restores the old allocation 
behaviour, keeping the more conservative estimate only for the available-memory 
API.

Regards,
  Felix


From: Guo, Shikai 
Sent: Thursday, July 14, 2022 11:21 PM
To: Kuehling, Felix ; amd-gfx@lists.freedesktop.org 

Cc: Phillips, Daniel ; Ji, Ruili ; 
Liu, Aaron 
Subject: RE: [PATCH] drm/amdkfd: Remove Align VRAM allocations to 1MB on APU 
ASIC

[AMD Official Use Only - General]

Thanks Felix comment, I will further debug this issue.

-Original Message-
From: Guo, Shikai
Sent: Friday, July 15, 2022 11:21 AM
To: Kuehling, Felix ; amd-gfx@lists.freedesktop.org
Cc: Phillips, Daniel ; Ji, Ruili ; 
Liu, Aaron 
Subject: RE: [PATCH] drm/amdkfd: Remove Align VRAM allocations to 1MB on APU 
ASIC

[AMD Official Use Only - General]

This Felix comment, I will further debug this issue.

-Original Message-
From: Kuehling, Felix 
Sent: Wednesday, July 13, 2022 10:17 PM
To: Guo, Shikai ; amd-gfx@lists.freedesktop.org
Cc: Phillips, Daniel ; Ji, Ruili ; 
Liu, Aaron 
Subject: Re: [PATCH] drm/amdkfd: Remove Align VRAM allocations to 1MB on APU 
ASIC


Am 2022-07-13 um 05:14 schrieb shikai guo:
> From: Shikai Guo 
>
> While executing KFDMemoryTest.MMBench, test case will allocate 4KB size 
> memory 1000 times.
> Every time, user space will get 2M memory.APU VRAM is 512M, there is not 
> enough memory to be allocated.
> So the 2M aligned feature is not suitable for APU.
NAK. We can try to make the estimate of available VRAM more accurate.
But in the end, this comes down to limitations of the VRAM manager and how it 
handles memory fragmentation.

A large discrepancy between total VRAM and available VRAM can have a few
reasons:

  * Big system memory means we need to reserve more space for page tables
  * Many small allocations causing lots of fragmentation. This may be
the result of memory leaks in previous tests

This patch can "fix" a situation where a leak caused excessive fragmentation. 
But that just papers over the leak. And it will cause the opposite problem for 
the new AvailableMemory test that checks that we can really allocate as much 
memory as we promised.

Regards,
   Felix


>
> guoshikai@guoshikai-MayanKD-RMB:~/linth/libhsakmt/tests/kfdtest/build$ 
> ./kfdtest --gtest_filter=KFDMemoryTest.MMBench
> [  ] Profile: Full Test
> [  ] HW capabilities: 0x9
> Note: Google Test filter = KFDMemoryTest.MMBench [==] Running
> 1 test from 1 test case.
> [--] Global test environment set-up.
> [--] 1 test from KFDMemoryTest
> [ RUN  ] KFDMemoryTest.MMBench
> [  ] Found VRAM of 512MB.
> [  ] Available VRAM 328MB.
> [  ] Test (avg. ns) alloc   mapOne  umapOne   mapAll  umapAll 
> free
> [  ] 
> --
> [  ]   4K-SysMem-noSDMA 2656110350 5212 3787 3981 
>12372
> [  ]  64K-SysMem-noSDMA 42864 6648 3973 5223 3843 
>15100
> [  ]   2M-SysMem-noSDMA31290612614 4390 6254 4790 
>70260
> [  ]  32M-SysMem-noSDMA   4417812   130437216259768718500 
>   929562
> [  ]   1G-SysMem-noSDMA 132161000  2738000   583000  2181000   499000 
> 39091000
> [  ] 
> --
> /home/guoshikai/linth/libhsakmt/tests/kfdtest/src/KFDMemoryTest.cpp:92
> 2: Failure Value of: (hsaKmtAllocMemory(allocNode, bufSize, memFlags, 
> &bufs[i]))
>Actual: 6
> Expected: HSAKMT_STATUS_SUCCESS
> Which is: 0
> [  FAILED  ] KFDMemoryTest.MMBench (749 ms)
>
> fix this issue by adding different treatments for apu and dgpu
>
> Signed-off-by: ruili ji 
> Signed-off-by: shikai guo 
> ---
>   .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c   | 18 +-
>   1 file changed, 13 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> index d1657de5f875..2ad2cd5e3e8b 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> @@ -115,7 +115,9 @@ void amdgpu_amdkfd_reserve_system_mem(uint64_t size)
>* compromise that should work in most cases without reserving too
>* much memory for page tables unnecessarily (factor 16K, >> 14).
>*/
> -#define ESTIMATE_PT_SIZE(mem_size) max(((mem_size) >> 14),
> AMDGPU_VM_RESERVED_VRAM)
> +
> +#define ESTIMATE_PT_SIZE(adev, mem_size)   (adev->flags & AMD_IS_APU) ? \
> +(mem_size >> 14) : max(((mem_size) >>

RE: [PATCH] drm/amdgpu: Ignore KFD eviction fences invalidating preemptible DMABuf imports

[Kuehling, Felix]

[AMD Official Use Only - General]

Re-mapping typically happens after evictions, before a new eviction fence gets 
attached. At that time the old eviction fence should be in the signaled state 
already, so it can't be signaled again. Therefore I would expect my patch to 
help with unmapping the DMABuf import, without breaking the eviction case.

Are you talking about remapping with a map-to-gpu call from user mode? I think 
that would only be a problem if the KFD BO was unmapped and remapped multiple 
times. The first time it's mapped, the fresh dmabuf import should be in the 
SYSTEM domain, so the validation in the SYSTEM domain before GTT would be a 
no-op.

I sort of agree that we don't really rely on the eviction fence on the DMABuf 
import. The reservation object is shared with the original BO. Moving the 
original BO triggers the eviction fence, so we don't need to trigger it again 
on the dmabuf import. Other than moving the original BO, I don't think we can 
do anything to the DMABuf import that would require an eviction for KFD use 
case. It is a special use case because we control both the import and the 
export in the same context.

In the general case dmabuf imports need their eviction fences. For example when 
we're importing a DMABuf from somewhere else, so the eviction fence is not 
shared with a BO that we already control. Even then, unmapping a dmabuf from 
our KFD VM does not need to wait for any fences on the DMABuf.

Regards,
  Felix

-Original Message-
From: Huang, JinHuiEric  
Sent: Thursday, April 27, 2023 14:58
To: Kuehling, Felix ; Koenig, Christian 
; Christian König ; 
amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH] drm/amdgpu: Ignore KFD eviction fences invalidating 
preemptible DMABuf imports

Hi Felix,

I tested your patch on mGPU systems. It doesn't break any KFD eviction tests, 
because tests don't allocate DMABuf import, that doesn't trigger it's eviction 
fence. The only thing the patch affects is in re-mapping DMABuf imports that 
the eviction will still be triggered.

I have an idea that we probably can remove eviction fence for GTT bo, because 
currently the only way to trigger the eviction fence is by calling 
ttm_bo_validate for CPU domain in kfd_mem_dmaunmap_dmabuf. Do you know there is 
other case to trigger GTT bo's eviction?

Regards,
Eric

On 2023-04-26 22:21, Felix Kuehling wrote:
> Hi Eric,
>
> Can you try if the attached patch fixes the problem without breaking 
> the eviction tests on a multi-GPU PCIe P2P system?
>
> Thanks,
>   Felix
>
>
> On 2023-04-26 13:02, Christian König wrote:
>> Am 26.04.23 um 18:58 schrieb Felix Kuehling:
>>>
>>> On 2023-04-26 9:03, Christian König wrote:
>>>> Am 25.04.23 um 16:11 schrieb Eric Huang:
>>>>> Hi Christian,
>>>>>
>>>>> What do you think about Felix's explanation?
>>>>
>>>> That's unfortunately not something we can do here.
>>>>
>>>>>
>>>>> Regards,
>>>>> Eric
>>>>>
>>>>> On 2023-04-13 09:28, Felix Kuehling wrote:
>>>>>> Am 2023-04-13 um 07:35 schrieb Christian König:
>>>>>>> Am 13.04.23 um 03:01 schrieb Felix Kuehling:
>>>>>>>> Am 2023-04-12 um 18:25 schrieb Eric Huang:
>>>>>>>>> It is to avoid redundant eviction for KFD's DMAbuf import bo 
>>>>>>>>> when dmaunmapping DMAbuf. The DMAbuf import bo has been set as 
>>>>>>>>> AMDGPU_PL_PREEMPT in KFD when mapping.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Eric Huang 
>>>>>>>>
>>>>>>>> Reviewed-by: Felix Kuehling 
>>>>>>>>
>>>>>>>> I'd like to get an Acked-by from Christian as well before 
>>>>>>>> submitting this.
>>>>>>>
>>>>>>> I have to admit that I only partially followed the internal 
>>>>>>> discussion, but in general you need a *really* good explanation 
>>>>>>> for this.
>>>>>>>
>>>>>>> E.g. add code comment and explain in the commit message 
>>>>>>> extensively why this is needed and why there are no alternatives.
>>>>>>
>>>>>> OK. I'll give it a shot:
>>>>>>
>>>>>>    This code path is used among other things when invalidating 
>>>>>> DMABuf
>>>>>>    imports. These imports share a reservation object with the 
>>>>>> exported
>>>>>>    BO. Waiting on all the fenc

Re: Stack out of bounds in KFD on Arcturus

[Kuehling, Felix]

I don't see why this problem would be specific to Arcturus. I don't see 
any excessive allocations on the stack either. Also the code involved 
here hasn't changed recently.

Are you using some weird kernel config with a smaller stack? Is it 
specific to a compiler version or some optimization flags? I've 
sometimes seen function inlining cause excessive stack usage.

Regards,
   Felix

On 2019-10-17 4:09 p.m., Grodzovsky, Andrey wrote:
> He Felix - I see this on boot when working with Arcturus.
>
> Andrey
>
>
> [  103.602092] kfd kfd: Allocated 3969056 bytes on gart
> [  103.610769]
> ==
> [  103.611469] BUG: KASAN: stack-out-of-bounds in
> kfd_create_vcrat_image_gpu+0x5db/0xb80 [amdgpu]
> [  103.611646] Read of size 4 at addr 8883cb19ee38 by task modprobe/1122
>
> [  103.611836] CPU: 3 PID: 1122 Comm: modprobe Tainted: G
> O  5.3.0-rc3+ #45
> [  103.611847] Hardware name: System manufacturer System Product
> Name/Z170-PRO, BIOS 1902 06/27/2016
> [  103.611856] Call Trace:
> [  103.611879]  dump_stack+0x71/0xab
> [  103.611907]  print_address_description+0x1da/0x3c0
> [  103.612453]  ? kfd_create_vcrat_image_gpu+0x5db/0xb80 [amdgpu]
> [  103.612479]  __kasan_report+0x13f/0x1a0
> [  103.613022]  ? kfd_create_vcrat_image_gpu+0x5db/0xb80 [amdgpu]
> [  103.613580]  ? kfd_create_vcrat_image_gpu+0x5db/0xb80 [amdgpu]
> [  103.613604]  kasan_report+0xe/0x20
> [  103.614149]  kfd_create_vcrat_image_gpu+0x5db/0xb80 [amdgpu]
> [  103.614762]  ? kfd_fill_gpu_memory_affinity+0x110/0x110 [amdgpu]
> [  103.614796]  ? __alloc_pages_nodemask+0x2c9/0x560
> [  103.614824]  ? __alloc_pages_slowpath+0x1390/0x1390
> [  103.614898]  ? kmalloc_order+0x63/0x70
> [  103.615469]  kfd_create_crat_image_virtual+0x70c/0x770 [amdgpu]
> [  103.616054]  ? kfd_create_crat_image_acpi+0x1c0/0x1c0 [amdgpu]
> [  103.616095]  ? up_write+0x4b/0x70
> [  103.616649]  kfd_topology_add_device+0x98d/0xb10 [amdgpu]
> [  103.617207]  ? kfd_topology_shutdown+0x60/0x60 [amdgpu]
> [  103.617743]  ? start_cpsch+0x2ff/0x3a0 [amdgpu]
> [  103.61]  ? mutex_lock_io_nested+0xac0/0xac0
> [  103.617807]  ? __mutex_unlock_slowpath+0xda/0x420
> [  103.617848]  ? __mutex_unlock_slowpath+0xda/0x420
> [  103.617877]  ? wait_for_completion+0x200/0x200
> [  103.618461]  ? start_cpsch+0x38b/0x3a0 [amdgpu]
> [  103.619011]  ? create_queue_cpsch+0x670/0x670 [amdgpu]
> [  103.619573]  ? kfd_iommu_device_init+0x92/0x1e0 [amdgpu]
> [  103.620112]  ? kfd_iommu_resume+0x2c/0x2c0 [amdgpu]
> [  103.620655]  ? kfd_iommu_check_device+0xf0/0xf0 [amdgpu]
> [  103.621228]  kgd2kfd_device_init+0x474/0x870 [amdgpu]
> [  103.621781]  amdgpu_amdkfd_device_init+0x291/0x390 [amdgpu]
> [  103.622329]  ? amdgpu_amdkfd_device_probe+0x90/0x90 [amdgpu]
> [  103.622344]  ? kmsg_dump_rewind_nolock+0x59/0x59
> [  103.622895]  ? amdgpu_ras_eeprom_test+0x71/0x90 [amdgpu]
> [  103.623424]  amdgpu_device_init+0x1bbe/0x2f00 [amdgpu]
> [  103.623819]  ? amdgpu_device_has_dc_support+0x30/0x30 [amdgpu]
> [  103.623842]  ? __isolate_free_page+0x290/0x290
> [  103.623852]  ? fs_reclaim_acquire.part.97+0x5/0x30
> [  103.623891]  ? __alloc_pages_nodemask+0x2c9/0x560
> [  103.623912]  ? __alloc_pages_slowpath+0x1390/0x1390
> [  103.623945]  ? kasan_unpoison_shadow+0x31/0x40
> [  103.623970]  ? kmalloc_order+0x63/0x70
> [  103.624337]  amdgpu_driver_load_kms+0xd9/0x430 [amdgpu]
> [  103.624690]  ? amdgpu_register_gpu_instance+0xe0/0xe0 [amdgpu]
> [  103.624756]  ? drm_dev_register+0x19c/0x310 [drm]
> [  103.624768]  ? __kasan_slab_free+0x133/0x160
> [  103.624849]  drm_dev_register+0x1f5/0x310 [drm]
> [  103.625212]  amdgpu_pci_probe+0x109/0x1f0 [amdgpu]
> [  103.625565]  ? amdgpu_pmops_runtime_idle+0xe0/0xe0 [amdgpu]
> [  103.625580]  local_pci_probe+0x74/0xd0
> [  103.625603]  pci_device_probe+0x1fa/0x310
> [  103.625620]  ? pci_device_remove+0x1c0/0x1c0
> [  103.625640]  ? sysfs_do_create_link_sd.isra.2+0x74/0xe0
> [  103.625673]  really_probe+0x367/0x5d0
> [  103.625700]  driver_probe_device+0x177/0x1b0
> [  103.625721]  device_driver_attach+0x8a/0x90
> [  103.625737]  ? device_driver_attach+0x90/0x90
> [  103.625746]  __driver_attach+0xeb/0x190
> [  103.625765]  ? device_driver_attach+0x90/0x90
> [  103.625773]  bus_for_each_dev+0xe4/0x160
> [  103.625789]  ? subsys_dev_iter_exit+0x10/0x10
> [  103.625829]  bus_add_driver+0x277/0x330
> [  103.625855]  driver_register+0xc6/0x1a0
> [  103.625866]  ? 0xa0d88000
> [  103.625880]  do_one_initcall+0xd3/0x334
> [  103.625895]  ? trace_event_raw_event_initcall_finish+0x150/0x150
> [  103.625911]  ? kasan_unpoison_shadow+0x31/0x40
> [  103.625924]  ? __kasan_kmalloc+0xd5/0xf0
> [  103.625946]  ? kmem_cache_alloc_trace+0x154/0x300
> [  103.625955]  ? kasan_unpoison_shadow+0x31/0x40
> [  103.625985]  do_init_module+0xec/0x354
> [  103.626011]  load_module+0x3c91/0x4980
> [  103.626118]  ? module_frob_arch_sections+0x20/0x20
> [  103.626132]  ? ima_read_file+0x10/0x10
> [  103.626142] 

Re: [PATCH] drm/amdkfd: kfd open return failed if device is locked

[Kuehling, Felix]

On 2019-10-18 10:27 a.m., Yang, Philip wrote:
> If device is locked for suspend and resume, kfd open should return
> failed -EAGAIN without creating process, otherwise the application exit
> to release the process will hang to wait for resume is done if the suspend
> and resume is stuck somewhere. This is backtrace:

This doesn't fix processes that were created before suspend/resume got 
stuck. They would still get stuck with the same backtrace. So this is 
jut a band-aid. The real underlying problem, that is not getting 
addressed, is suspend/resume getting stuck.

Am I missing something?

Regards,
   Felix


>
> [Thu Oct 17 16:43:37 2019] INFO: task rocminfo:3024 blocked for more
> than 120 seconds.
> [Thu Oct 17 16:43:37 2019]   Not tainted
> 5.0.0-rc1-kfd-compute-rocm-dkms-no-npi-1131 #1
> [Thu Oct 17 16:43:37 2019] "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [Thu Oct 17 16:43:37 2019] rocminfoD0  3024   2947
> 0x8000
> [Thu Oct 17 16:43:37 2019] Call Trace:
> [Thu Oct 17 16:43:37 2019]  ? __schedule+0x3d9/0x8a0
> [Thu Oct 17 16:43:37 2019]  schedule+0x32/0x70
> [Thu Oct 17 16:43:37 2019]  schedule_preempt_disabled+0xa/0x10
> [Thu Oct 17 16:43:37 2019]  __mutex_lock.isra.9+0x1e3/0x4e0
> [Thu Oct 17 16:43:37 2019]  ? __call_srcu+0x264/0x3b0
> [Thu Oct 17 16:43:37 2019]  ? process_termination_cpsch+0x24/0x2f0
> [amdgpu]
> [Thu Oct 17 16:43:37 2019]  process_termination_cpsch+0x24/0x2f0
> [amdgpu]
> [Thu Oct 17 16:43:37 2019]
> kfd_process_dequeue_from_all_devices+0x42/0x60 [amdgpu]
> [Thu Oct 17 16:43:37 2019]  kfd_process_notifier_release+0x1be/0x220
> [amdgpu]
> [Thu Oct 17 16:43:37 2019]  __mmu_notifier_release+0x3e/0xc0
> [Thu Oct 17 16:43:37 2019]  exit_mmap+0x160/0x1a0
> [Thu Oct 17 16:43:37 2019]  ? __handle_mm_fault+0xba3/0x1200
> [Thu Oct 17 16:43:37 2019]  ? exit_robust_list+0x5a/0x110
> [Thu Oct 17 16:43:37 2019]  mmput+0x4a/0x120
> [Thu Oct 17 16:43:37 2019]  do_exit+0x284/0xb20
> [Thu Oct 17 16:43:37 2019]  ? handle_mm_fault+0xfa/0x200
> [Thu Oct 17 16:43:37 2019]  do_group_exit+0x3a/0xa0
> [Thu Oct 17 16:43:37 2019]  __x64_sys_exit_group+0x14/0x20
> [Thu Oct 17 16:43:37 2019]  do_syscall_64+0x4f/0x100
> [Thu Oct 17 16:43:37 2019]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> Signed-off-by: Philip Yang 
> ---
>   drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 6 +++---
>   1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> index d9e36dbf13d5..40d75c39f08e 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> @@ -120,13 +120,13 @@ static int kfd_open(struct inode *inode, struct file 
> *filep)
>   return -EPERM;
>   }
>   
> + if (kfd_is_locked())
> + return -EAGAIN;
> +
>   process = kfd_create_process(filep);
>   if (IS_ERR(process))
>   return PTR_ERR(process);
>   
> - if (kfd_is_locked())
> - return -EAGAIN;
> -
>   dev_dbg(kfd_device, "process %d opened, compat mode (32 bit) - %d\n",
>   process->pasid, process->is_32bit_user_mode);
>   
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH v2] drm/amdkfd: kfd open return failed if device is locked

[Kuehling, Felix]

On 2019-10-18 1:36 p.m., Yang, Philip wrote:
> If device is locked for suspend and resume, kfd open should return
> failed -EAGAIN without creating process, otherwise the application exit
> to release the process will hang to wait for resume is done if the suspend
> and resume is stuck somewhere. This is backtrace:
>
> v2: fix processes that were created before suspend/resume got stuck
>
> [Thu Oct 17 16:43:37 2019] INFO: task rocminfo:3024 blocked for more
> than 120 seconds.
> [Thu Oct 17 16:43:37 2019]   Not tainted
> 5.0.0-rc1-kfd-compute-rocm-dkms-no-npi-1131 #1
> [Thu Oct 17 16:43:37 2019] "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [Thu Oct 17 16:43:37 2019] rocminfoD0  3024   2947
> 0x8000
> [Thu Oct 17 16:43:37 2019] Call Trace:
> [Thu Oct 17 16:43:37 2019]  ? __schedule+0x3d9/0x8a0
> [Thu Oct 17 16:43:37 2019]  schedule+0x32/0x70
> [Thu Oct 17 16:43:37 2019]  schedule_preempt_disabled+0xa/0x10
> [Thu Oct 17 16:43:37 2019]  __mutex_lock.isra.9+0x1e3/0x4e0
> [Thu Oct 17 16:43:37 2019]  ? __call_srcu+0x264/0x3b0
> [Thu Oct 17 16:43:37 2019]  ? process_termination_cpsch+0x24/0x2f0
> [amdgpu]
> [Thu Oct 17 16:43:37 2019]  process_termination_cpsch+0x24/0x2f0
> [amdgpu]
> [Thu Oct 17 16:43:37 2019]
> kfd_process_dequeue_from_all_devices+0x42/0x60 [amdgpu]
> [Thu Oct 17 16:43:37 2019]  kfd_process_notifier_release+0x1be/0x220
> [amdgpu]
> [Thu Oct 17 16:43:37 2019]  __mmu_notifier_release+0x3e/0xc0
> [Thu Oct 17 16:43:37 2019]  exit_mmap+0x160/0x1a0
> [Thu Oct 17 16:43:37 2019]  ? __handle_mm_fault+0xba3/0x1200
> [Thu Oct 17 16:43:37 2019]  ? exit_robust_list+0x5a/0x110
> [Thu Oct 17 16:43:37 2019]  mmput+0x4a/0x120
> [Thu Oct 17 16:43:37 2019]  do_exit+0x284/0xb20
> [Thu Oct 17 16:43:37 2019]  ? handle_mm_fault+0xfa/0x200
> [Thu Oct 17 16:43:37 2019]  do_group_exit+0x3a/0xa0
> [Thu Oct 17 16:43:37 2019]  __x64_sys_exit_group+0x14/0x20
> [Thu Oct 17 16:43:37 2019]  do_syscall_64+0x4f/0x100
> [Thu Oct 17 16:43:37 2019]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> Signed-off-by: Philip Yang 
> ---
>   drivers/gpu/drm/amd/amdkfd/kfd_chardev.c   | 6 +++---
>   drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c | 6 ++
>   2 files changed, 9 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> index d9e36dbf13d5..40d75c39f08e 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> @@ -120,13 +120,13 @@ static int kfd_open(struct inode *inode, struct file 
> *filep)
>   return -EPERM;
>   }
>   
> + if (kfd_is_locked())
> + return -EAGAIN;
> +
>   process = kfd_create_process(filep);
>   if (IS_ERR(process))
>   return PTR_ERR(process);
>   
> - if (kfd_is_locked())
> - return -EAGAIN;
> -
>   dev_dbg(kfd_device, "process %d opened, compat mode (32 bit) - %d\n",
>   process->pasid, process->is_32bit_user_mode);
>   
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c
> index 8509814a6ff0..3784013b92a0 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c
> @@ -128,6 +128,12 @@ void kfd_process_dequeue_from_all_devices(struct 
> kfd_process *p)
>   {
>   struct kfd_process_device *pdd;
>   
> + /* If suspend/resume got stuck, dqm_lock is hold,
> +  * skip process_termination_cpsch to avoid deadlock
> +  */
> + if (kfd_is_locked())
> + return;
> +

Holding the DQM lock during reset has caused other problems (lock 
dependency issues and deadlocks) and I was thinking about getting rid of 
that completely. The intention of holding the DQM lock during reset was 
to prevent the device queue manager from accessing the CP hardware while 
a reset was in progress. However, I think there are smarter ways to 
achieve that. We already get a pre-reset callback (kgd2kfd_pre_reset) 
that executes the kgd2kfd_suspend, which suspends processes and stops 
DQM through kfd->dqm->ops.stop(kfd->dqm). This should take care of most 
of the problem. If there are any places in DQM that try to access the 
devices, they should add conditions to not access HW while DQM is 
stopped. Then we could avoid holding a lock indefinitely while a reset 
is in progress.

The DQM lock is particularly problematic in terms of lock dependencies 
because it can be taken in MMU notifiers. We want to avoid taking any 
other locks while holding the DQM lock. Holding the DQM lock for a long 
time during reset is counterproductive to that objective.

Regards,
   Felix


>   list_for_each_entry(pdd, &p->per_device_data, per_device_list)
>   kfd_process_dequeue_from_device(pdd);
>   }
___
amd-gfx mailing list
amd-gfx@lists.freedesk

Re: [PATCH 2/2] Revert "drm/amdgpu: disable c-states on xgmi perfmons"

[Kuehling, Felix]

You can squash the two reverts into a single commit so you avoid 
reintroducing a broken intermediate state. Mention both reverted commits 
in the squashed commit description. Checkpatch.pl prefers a different 
format for quoting reverted commits. Run checkpatch.pl on your commit to 
see a proper example.

Regards,
   Felix


On 2019-10-18 1:59 p.m., Kim, Jonathan wrote:
> This reverts commit 54275cd1649f4034c6450b6c5a8358fcd4f7dda6.
>
> incomplete solution to df c-state race condition.  smu msg in perf events
> causes deadlock.
>
> Change-Id: Ia85179df2bd167657e42a2d828c4a7c475c392ff
> Signed-off-by: Jonathan Kim 
> ---
>   drivers/gpu/drm/amd/amdgpu/df_v3_6.c | 36 +---
>   1 file changed, 1 insertion(+), 35 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/df_v3_6.c 
> b/drivers/gpu/drm/amd/amdgpu/df_v3_6.c
> index f403c62c944e..16fbd2bc8ad1 100644
> --- a/drivers/gpu/drm/amd/amdgpu/df_v3_6.c
> +++ b/drivers/gpu/drm/amd/amdgpu/df_v3_6.c
> @@ -93,21 +93,6 @@ const struct attribute_group *df_v3_6_attr_groups[] = {
>   NULL
>   };
>   
> -static df_v3_6_set_df_cstate(struct amdgpu_device *adev, int allow)
> -{
> - int r = 0;
> -
> - if (is_support_sw_smu(adev)) {
> - r = smu_set_df_cstate(&adev->smu, allow);
> - } else if (adev->powerplay.pp_funcs
> - && adev->powerplay.pp_funcs->set_df_cstate) {
> - r = adev->powerplay.pp_funcs->set_df_cstate(
> - adev->powerplay.pp_handle, allow);
> - }
> -
> - return r;
> -}
> -
>   static uint64_t df_v3_6_get_fica(struct amdgpu_device *adev,
>uint32_t ficaa_val)
>   {
> @@ -117,9 +102,6 @@ static uint64_t df_v3_6_get_fica(struct amdgpu_device 
> *adev,
>   address = adev->nbio.funcs->get_pcie_index_offset(adev);
>   data = adev->nbio.funcs->get_pcie_data_offset(adev);
>   
> - if (df_v3_6_set_df_cstate(adev, DF_CSTATE_DISALLOW))
> - return 0x;
> -
>   spin_lock_irqsave(&adev->pcie_idx_lock, flags);
>   WREG32(address, smnDF_PIE_AON_FabricIndirectConfigAccessAddress3);
>   WREG32(data, ficaa_val);
> @@ -132,8 +114,6 @@ static uint64_t df_v3_6_get_fica(struct amdgpu_device 
> *adev,
>   
>   spin_unlock_irqrestore(&adev->pcie_idx_lock, flags);
>   
> - df_v3_6_set_df_cstate(adev, DF_CSTATE_ALLOW);
> -
>   return (((ficadh_val & 0x) << 32) | ficadl_val);
>   }
>   
> @@ -145,9 +125,6 @@ static void df_v3_6_set_fica(struct amdgpu_device *adev, 
> uint32_t ficaa_val,
>   address = adev->nbio.funcs->get_pcie_index_offset(adev);
>   data = adev->nbio.funcs->get_pcie_data_offset(adev);
>   
> - if (df_v3_6_set_df_cstate(adev, DF_CSTATE_DISALLOW))
> - return;
> -
>   spin_lock_irqsave(&adev->pcie_idx_lock, flags);
>   WREG32(address, smnDF_PIE_AON_FabricIndirectConfigAccessAddress3);
>   WREG32(data, ficaa_val);
> @@ -157,9 +134,8 @@ static void df_v3_6_set_fica(struct amdgpu_device *adev, 
> uint32_t ficaa_val,
>   
>   WREG32(address, smnDF_PIE_AON_FabricIndirectConfigAccessDataHi3);
>   WREG32(data, ficadh_val);
> - spin_unlock_irqrestore(&adev->pcie_idx_lock, flags);
>   
> - df_v3_6_set_df_cstate(adev, DF_CSTATE_ALLOW);
> + spin_unlock_irqrestore(&adev->pcie_idx_lock, flags);
>   }
>   
>   /*
> @@ -177,17 +153,12 @@ static void df_v3_6_perfmon_rreg(struct amdgpu_device 
> *adev,
>   address = adev->nbio.funcs->get_pcie_index_offset(adev);
>   data = adev->nbio.funcs->get_pcie_data_offset(adev);
>   
> - if (df_v3_6_set_df_cstate(adev, DF_CSTATE_DISALLOW))
> - return;
> -
>   spin_lock_irqsave(&adev->pcie_idx_lock, flags);
>   WREG32(address, lo_addr);
>   *lo_val = RREG32(data);
>   WREG32(address, hi_addr);
>   *hi_val = RREG32(data);
>   spin_unlock_irqrestore(&adev->pcie_idx_lock, flags);
> -
> - df_v3_6_set_df_cstate(adev, DF_CSTATE_ALLOW);
>   }
>   
>   /*
> @@ -204,17 +175,12 @@ static void df_v3_6_perfmon_wreg(struct amdgpu_device 
> *adev, uint32_t lo_addr,
>   address = adev->nbio.funcs->get_pcie_index_offset(adev);
>   data = adev->nbio.funcs->get_pcie_data_offset(adev);
>   
> - if (df_v3_6_set_df_cstate(adev, DF_CSTATE_DISALLOW))
> - return;
> -
>   spin_lock_irqsave(&adev->pcie_idx_lock, flags);
>   WREG32(address, lo_addr);
>   WREG32(data, lo_val);
>   WREG32(address, hi_addr);
>   WREG32(data, hi_val);
>   spin_unlock_irqrestore(&adev->pcie_idx_lock, flags);
> -
> - df_v3_6_set_df_cstate(adev, DF_CSTATE_ALLOW);
>   }
>   
>   /* get the number of df counters available */
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdgpu: revert calling smu msg in df callbacks

[Kuehling, Felix]

On 2019-10-18 4:29 p.m., Kim, Jonathan wrote:
> reverting the following changes:
> commit 7dd2eb31fcd5 ("drm/amdgpu: fix compiler warnings for df perfmons")
> commit 54275cd1649f ("drm/amdgpu: disable c-states on xgmi perfmons")
>
> perf events use spin-locks.  embedded smu messages have potential long
> response times and potentially deadlocks the system.
>
> Change-Id: Ic36c35a62dec116d0a2f5b69c22af4d414458679
> Signed-off-by: Jonathan Kim 

Reviewed-by: Felix Kuehling 

See one more comment inline below ...


> ---
>   drivers/gpu/drm/amd/amdgpu/df_v3_6.c | 38 ++--
>   1 file changed, 2 insertions(+), 36 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/df_v3_6.c 
> b/drivers/gpu/drm/amd/amdgpu/df_v3_6.c
> index e1cf7e9c616a..16fbd2bc8ad1 100644
> --- a/drivers/gpu/drm/amd/amdgpu/df_v3_6.c
> +++ b/drivers/gpu/drm/amd/amdgpu/df_v3_6.c
> @@ -93,21 +93,6 @@ const struct attribute_group *df_v3_6_attr_groups[] = {
>   NULL
>   };
>   
> -static int df_v3_6_set_df_cstate(struct amdgpu_device *adev, int allow)
> -{
> - int r = 0;
> -
> - if (is_support_sw_smu(adev)) {
> - r = smu_set_df_cstate(&adev->smu, allow);
> - } else if (adev->powerplay.pp_funcs
> - && adev->powerplay.pp_funcs->set_df_cstate) {
> - r = adev->powerplay.pp_funcs->set_df_cstate(
> - adev->powerplay.pp_handle, allow);
> - }
> -
> - return r;
> -}
> -
>   static uint64_t df_v3_6_get_fica(struct amdgpu_device *adev,
>uint32_t ficaa_val)
>   {
> @@ -117,9 +102,6 @@ static uint64_t df_v3_6_get_fica(struct amdgpu_device 
> *adev,
>   address = adev->nbio.funcs->get_pcie_index_offset(adev);
>   data = adev->nbio.funcs->get_pcie_data_offset(adev);
>   
> - if (df_v3_6_set_df_cstate(adev, DF_CSTATE_DISALLOW))
> - return 0x;
> -
>   spin_lock_irqsave(&adev->pcie_idx_lock, flags);
>   WREG32(address, smnDF_PIE_AON_FabricIndirectConfigAccessAddress3);
>   WREG32(data, ficaa_val);
> @@ -132,8 +114,6 @@ static uint64_t df_v3_6_get_fica(struct amdgpu_device 
> *adev,
>   
>   spin_unlock_irqrestore(&adev->pcie_idx_lock, flags);
>   
> - df_v3_6_set_df_cstate(adev, DF_CSTATE_ALLOW);
> -
>   return (((ficadh_val & 0x) << 32) | ficadl_val);
>   }
>   
> @@ -145,9 +125,6 @@ static void df_v3_6_set_fica(struct amdgpu_device *adev, 
> uint32_t ficaa_val,
>   address = adev->nbio.funcs->get_pcie_index_offset(adev);
>   data = adev->nbio.funcs->get_pcie_data_offset(adev);
>   
> - if (df_v3_6_set_df_cstate(adev, DF_CSTATE_DISALLOW))
> - return;
> -
>   spin_lock_irqsave(&adev->pcie_idx_lock, flags);
>   WREG32(address, smnDF_PIE_AON_FabricIndirectConfigAccessAddress3);
>   WREG32(data, ficaa_val);
> @@ -157,9 +134,8 @@ static void df_v3_6_set_fica(struct amdgpu_device *adev, 
> uint32_t ficaa_val,
>   
>   WREG32(address, smnDF_PIE_AON_FabricIndirectConfigAccessDataHi3);
>   WREG32(data, ficadh_val);
> - spin_unlock_irqrestore(&adev->pcie_idx_lock, flags);
>   
> - df_v3_6_set_df_cstate(adev, DF_CSTATE_ALLOW);
> + spin_unlock_irqrestore(&adev->pcie_idx_lock, flags);
>   }
>   
>   /*
> @@ -177,17 +153,12 @@ static void df_v3_6_perfmon_rreg(struct amdgpu_device 
> *adev,
>   address = adev->nbio.funcs->get_pcie_index_offset(adev);
>   data = adev->nbio.funcs->get_pcie_data_offset(adev);
>   
> - if (df_v3_6_set_df_cstate(adev, DF_CSTATE_DISALLOW))
> - return;
> -
>   spin_lock_irqsave(&adev->pcie_idx_lock, flags);
>   WREG32(address, lo_addr);
>   *lo_val = RREG32(data);
>   WREG32(address, hi_addr);
>   *hi_val = RREG32(data);
>   spin_unlock_irqrestore(&adev->pcie_idx_lock, flags);
> -
> - df_v3_6_set_df_cstate(adev, DF_CSTATE_ALLOW);
>   }
>   
>   /*
> @@ -204,17 +175,12 @@ static void df_v3_6_perfmon_wreg(struct amdgpu_device 
> *adev, uint32_t lo_addr,
>   address = adev->nbio.funcs->get_pcie_index_offset(adev);
>   data = adev->nbio.funcs->get_pcie_data_offset(adev);
>   
> - if (df_v3_6_set_df_cstate(adev, DF_CSTATE_DISALLOW))
> - return;
> -
>   spin_lock_irqsave(&adev->pcie_idx_lock, flags);
>   WREG32(address, lo_addr);
>   WREG32(data, lo_val);
>   WREG32(address, hi_addr);
>   WREG32(data, hi_val);
>   spin_unlock_irqrestore(&adev->pcie_idx_lock, flags);
> -
> - df_v3_6_set_df_cstate(adev, DF_CSTATE_ALLOW);
>   }
>   
>   /* get the number of df counters available */
> @@ -546,7 +512,7 @@ static void df_v3_6_pmc_get_count(struct amdgpu_device 
> *adev,
> uint64_t config,
> uint64_t *count)
>   {
> - uint32_t lo_base_addr, hi_base_addr, lo_val = 0, hi_val = 0;
> + uint32_t lo_base_addr, hi_base_addr, lo_val, hi_val;

This part looks like it was unrelated to the DF Cstate changes. If this 

Re: Stack out of bounds in KFD on Arcturus

[Kuehling, Felix]

On 2019-10-17 6:38 p.m., Grodzovsky, Andrey wrote:
> Not that I aware of, is there a special Kconfig flag to determine stack
> size ?

I remember there used to be a Kconfig option to force a 4KB kernel 
stack. I don't see it in the current kernel any more.

I don't have time to work on this myself. I'll create a ticket and see 
if I can find someone to investigate.

Thanks,
   Felix


>
> Andrey
>
> On 10/17/19 5:29 PM, Kuehling, Felix wrote:
>> I don't see why this problem would be specific to Arcturus. I don't see
>> any excessive allocations on the stack either. Also the code involved
>> here hasn't changed recently.
>>
>> Are you using some weird kernel config with a smaller stack? Is it
>> specific to a compiler version or some optimization flags? I've
>> sometimes seen function inlining cause excessive stack usage.
>>
>> Regards,
>>  Felix
>>
>> On 2019-10-17 4:09 p.m., Grodzovsky, Andrey wrote:
>>> He Felix - I see this on boot when working with Arcturus.
>>>
>>> Andrey
>>>
>>>
>>> [  103.602092] kfd kfd: Allocated 3969056 bytes on gart
>>> [  103.610769]
>>> ==
>>> [  103.611469] BUG: KASAN: stack-out-of-bounds in
>>> kfd_create_vcrat_image_gpu+0x5db/0xb80 [amdgpu]
>>> [  103.611646] Read of size 4 at addr 8883cb19ee38 by task modprobe/1122
>>>
>>> [  103.611836] CPU: 3 PID: 1122 Comm: modprobe Tainted: G
>>> O  5.3.0-rc3+ #45
>>> [  103.611847] Hardware name: System manufacturer System Product
>>> Name/Z170-PRO, BIOS 1902 06/27/2016
>>> [  103.611856] Call Trace:
>>> [  103.611879]  dump_stack+0x71/0xab
>>> [  103.611907]  print_address_description+0x1da/0x3c0
>>> [  103.612453]  ? kfd_create_vcrat_image_gpu+0x5db/0xb80 [amdgpu]
>>> [  103.612479]  __kasan_report+0x13f/0x1a0
>>> [  103.613022]  ? kfd_create_vcrat_image_gpu+0x5db/0xb80 [amdgpu]
>>> [  103.613580]  ? kfd_create_vcrat_image_gpu+0x5db/0xb80 [amdgpu]
>>> [  103.613604]  kasan_report+0xe/0x20
>>> [  103.614149]  kfd_create_vcrat_image_gpu+0x5db/0xb80 [amdgpu]
>>> [  103.614762]  ? kfd_fill_gpu_memory_affinity+0x110/0x110 [amdgpu]
>>> [  103.614796]  ? __alloc_pages_nodemask+0x2c9/0x560
>>> [  103.614824]  ? __alloc_pages_slowpath+0x1390/0x1390
>>> [  103.614898]  ? kmalloc_order+0x63/0x70
>>> [  103.615469]  kfd_create_crat_image_virtual+0x70c/0x770 [amdgpu]
>>> [  103.616054]  ? kfd_create_crat_image_acpi+0x1c0/0x1c0 [amdgpu]
>>> [  103.616095]  ? up_write+0x4b/0x70
>>> [  103.616649]  kfd_topology_add_device+0x98d/0xb10 [amdgpu]
>>> [  103.617207]  ? kfd_topology_shutdown+0x60/0x60 [amdgpu]
>>> [  103.617743]  ? start_cpsch+0x2ff/0x3a0 [amdgpu]
>>> [  103.61]  ? mutex_lock_io_nested+0xac0/0xac0
>>> [  103.617807]  ? __mutex_unlock_slowpath+0xda/0x420
>>> [  103.617848]  ? __mutex_unlock_slowpath+0xda/0x420
>>> [  103.617877]  ? wait_for_completion+0x200/0x200
>>> [  103.618461]  ? start_cpsch+0x38b/0x3a0 [amdgpu]
>>> [  103.619011]  ? create_queue_cpsch+0x670/0x670 [amdgpu]
>>> [  103.619573]  ? kfd_iommu_device_init+0x92/0x1e0 [amdgpu]
>>> [  103.620112]  ? kfd_iommu_resume+0x2c/0x2c0 [amdgpu]
>>> [  103.620655]  ? kfd_iommu_check_device+0xf0/0xf0 [amdgpu]
>>> [  103.621228]  kgd2kfd_device_init+0x474/0x870 [amdgpu]
>>> [  103.621781]  amdgpu_amdkfd_device_init+0x291/0x390 [amdgpu]
>>> [  103.622329]  ? amdgpu_amdkfd_device_probe+0x90/0x90 [amdgpu]
>>> [  103.622344]  ? kmsg_dump_rewind_nolock+0x59/0x59
>>> [  103.622895]  ? amdgpu_ras_eeprom_test+0x71/0x90 [amdgpu]
>>> [  103.623424]  amdgpu_device_init+0x1bbe/0x2f00 [amdgpu]
>>> [  103.623819]  ? amdgpu_device_has_dc_support+0x30/0x30 [amdgpu]
>>> [  103.623842]  ? __isolate_free_page+0x290/0x290
>>> [  103.623852]  ? fs_reclaim_acquire.part.97+0x5/0x30
>>> [  103.623891]  ? __alloc_pages_nodemask+0x2c9/0x560
>>> [  103.623912]  ? __alloc_pages_slowpath+0x1390/0x1390
>>> [  103.623945]  ? kasan_unpoison_shadow+0x31/0x40
>>> [  103.623970]  ? kmalloc_order+0x63/0x70
>>> [  103.624337]  amdgpu_driver_load_kms+0xd9/0x430 [amdgpu]
>>> [  103.624690]  ? amdgpu_register_gpu_instance+0xe0/0xe0 [amdgpu]
>>> [  103.624756]  ? drm_dev_register+0x19c/0x310 [drm]
>>> [  103.624768]  ? __kasan_slab_free+0x133/0x160
>>> [  103.624849]  drm_dev_register+0x1f5/0x310 [drm]
>>> [  103.625212]  amdgpu_pci_probe+0x109/0x1f0 [amdgpu]
>

Re: [PATCH v2] drm/amdkfd: kfd open return failed if device is locked

[Kuehling, Felix]

On 2019-10-18 6:54 p.m., Zeng, Oak wrote:
> In current implementation, even dqm is stopped, user can still create (and 
> start) new queue. This is not correct. We should forbid user create/start new 
> queue if dqm is stopped - stop means stopping the current executing queues 
> and stop receiving new creating request.

Queues being stopped should be no reason not to create new queues. 
Creating a new queue is just allocating a new MQD and populating it. If 
the process is suspended (which it is during reset and suspend and 
evictions), there is no need to touch hardware registers or send an 
updated runlist to the HWS.

When the process is resumed at the end of the reset/suspend/eviction, 
that's when any newly created queues would get mapped to the hardware.

Regards,
   Felix


>
> Regards,
> Oak
>
> -Original Message-
> From: amd-gfx  On Behalf Of Kuehling, 
> Felix
> Sent: Friday, October 18, 2019 3:08 PM
> To: amd-gfx@lists.freedesktop.org; Yang, Philip 
> Subject: Re: [PATCH v2] drm/amdkfd: kfd open return failed if device is locked
>
> On 2019-10-18 1:36 p.m., Yang, Philip wrote:
>> If device is locked for suspend and resume, kfd open should return
>> failed -EAGAIN without creating process, otherwise the application
>> exit to release the process will hang to wait for resume is done if
>> the suspend and resume is stuck somewhere. This is backtrace:
>>
>> v2: fix processes that were created before suspend/resume got stuck
>>
>> [Thu Oct 17 16:43:37 2019] INFO: task rocminfo:3024 blocked for more
>> than 120 seconds.
>> [Thu Oct 17 16:43:37 2019]   Not tainted
>> 5.0.0-rc1-kfd-compute-rocm-dkms-no-npi-1131 #1 [Thu Oct 17 16:43:37
>> 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> [Thu Oct 17 16:43:37 2019] rocminfoD0  3024   2947
>> 0x8000
>> [Thu Oct 17 16:43:37 2019] Call Trace:
>> [Thu Oct 17 16:43:37 2019]  ? __schedule+0x3d9/0x8a0 [Thu Oct 17
>> 16:43:37 2019]  schedule+0x32/0x70 [Thu Oct 17 16:43:37 2019]
>> schedule_preempt_disabled+0xa/0x10
>> [Thu Oct 17 16:43:37 2019]  __mutex_lock.isra.9+0x1e3/0x4e0 [Thu Oct
>> 17 16:43:37 2019]  ? __call_srcu+0x264/0x3b0 [Thu Oct 17 16:43:37
>> 2019]  ? process_termination_cpsch+0x24/0x2f0
>> [amdgpu]
>> [Thu Oct 17 16:43:37 2019]  process_termination_cpsch+0x24/0x2f0
>> [amdgpu]
>> [Thu Oct 17 16:43:37 2019]
>> kfd_process_dequeue_from_all_devices+0x42/0x60 [amdgpu] [Thu Oct 17
>> 16:43:37 2019]  kfd_process_notifier_release+0x1be/0x220
>> [amdgpu]
>> [Thu Oct 17 16:43:37 2019]  __mmu_notifier_release+0x3e/0xc0 [Thu Oct
>> 17 16:43:37 2019]  exit_mmap+0x160/0x1a0 [Thu Oct 17 16:43:37 2019]  ?
>> __handle_mm_fault+0xba3/0x1200 [Thu Oct 17 16:43:37 2019]  ?
>> exit_robust_list+0x5a/0x110 [Thu Oct 17 16:43:37 2019]
>> mmput+0x4a/0x120 [Thu Oct 17 16:43:37 2019]  do_exit+0x284/0xb20 [Thu
>> Oct 17 16:43:37 2019]  ? handle_mm_fault+0xfa/0x200 [Thu Oct 17
>> 16:43:37 2019]  do_group_exit+0x3a/0xa0 [Thu Oct 17 16:43:37 2019]
>> __x64_sys_exit_group+0x14/0x20 [Thu Oct 17 16:43:37 2019]
>> do_syscall_64+0x4f/0x100 [Thu Oct 17 16:43:37 2019]
>> entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>
>> Signed-off-by: Philip Yang 
>> ---
>>drivers/gpu/drm/amd/amdkfd/kfd_chardev.c   | 6 +++---
>>drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c | 6 ++
>>2 files changed, 9 insertions(+), 3 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
>> b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
>> index d9e36dbf13d5..40d75c39f08e 100644
>> --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
>> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
>> @@ -120,13 +120,13 @@ static int kfd_open(struct inode *inode, struct file 
>> *filep)
>>  return -EPERM;
>>  }
>>
>> +if (kfd_is_locked())
>> +return -EAGAIN;
>> +
>>  process = kfd_create_process(filep);
>>  if (IS_ERR(process))
>>  return PTR_ERR(process);
>>
>> -if (kfd_is_locked())
>> -return -EAGAIN;
>> -
>>  dev_dbg(kfd_device, "process %d opened, compat mode (32 bit) - %d\n",
>>  process->pasid, process->is_32bit_user_mode);
>>
>> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c
>> b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c
>> index 8509814a6ff0..3784013b92a0 100644
>> --- a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c
>> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_

Re: [PATCH] drm/amdkfd: don't use dqm lock during device reset/suspend/resume

[Kuehling, Felix]

On 2019-10-21 5:04 p.m., Yang, Philip wrote:
> If device reset/suspend/resume failed for some reason, dqm lock is
> hold forever and this causes deadlock. Below is a kernel backtrace when
> application open kfd after suspend/resume failed.
>
> Instead of holding dqm lock in pre_reset and releasing dqm lock in
> post_reset, add dqm->device_stopped flag which is modified in
> dqm->ops.start and dqm->ops.stop. The flag doesn't need lock protection
> because write/read are all inside dqm lock.
>
> For HWS case, map_queues_cpsch and unmap_queues_cpsch checks
> device_stopped flag before sending the updated runlist.

What about the non-HWS case?

In theory in non-HWS case new queues should be created in evicted state 
while the device (and all processes) are suspended. So we should never 
try to map or unmap queues to HQDs during suspend. But I'd feel better 
with a WARN_ON and error return in the right places to make sure we're 
not missing anything. Basically, we can't call any of the 
load/destroy_mqd functions while suspended.

That reminds me, we also have to add some checks in the debugfs code to 
avoid dumping HQDs of a DQM that's stopped.

Last comment: dqm->device_stopped must be initialized as true. It will 
get set to false when the device is first started. It may be easier to 
reverse the logic, something like dqm->sched_running that gets 
implicitly initialized as false.

Regards,
   Felix


>
> Backtrace of dqm lock deadlock:
>
> [Thu Oct 17 16:43:37 2019] INFO: task rocminfo:3024 blocked for more
> than 120 seconds.
> [Thu Oct 17 16:43:37 2019]   Not tainted
> 5.0.0-rc1-kfd-compute-rocm-dkms-no-npi-1131 #1
> [Thu Oct 17 16:43:37 2019] "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [Thu Oct 17 16:43:37 2019] rocminfoD0  3024   2947
> 0x8000
> [Thu Oct 17 16:43:37 2019] Call Trace:
> [Thu Oct 17 16:43:37 2019]  ? __schedule+0x3d9/0x8a0
> [Thu Oct 17 16:43:37 2019]  schedule+0x32/0x70
> [Thu Oct 17 16:43:37 2019]  schedule_preempt_disabled+0xa/0x10
> [Thu Oct 17 16:43:37 2019]  __mutex_lock.isra.9+0x1e3/0x4e0
> [Thu Oct 17 16:43:37 2019]  ? __call_srcu+0x264/0x3b0
> [Thu Oct 17 16:43:37 2019]  ? process_termination_cpsch+0x24/0x2f0
> [amdgpu]
> [Thu Oct 17 16:43:37 2019]  process_termination_cpsch+0x24/0x2f0
> [amdgpu]
> [Thu Oct 17 16:43:37 2019]
> kfd_process_dequeue_from_all_devices+0x42/0x60 [amdgpu]
> [Thu Oct 17 16:43:37 2019]  kfd_process_notifier_release+0x1be/0x220
> [amdgpu]
> [Thu Oct 17 16:43:37 2019]  __mmu_notifier_release+0x3e/0xc0
> [Thu Oct 17 16:43:37 2019]  exit_mmap+0x160/0x1a0
> [Thu Oct 17 16:43:37 2019]  ? __handle_mm_fault+0xba3/0x1200
> [Thu Oct 17 16:43:37 2019]  ? exit_robust_list+0x5a/0x110
> [Thu Oct 17 16:43:37 2019]  mmput+0x4a/0x120
> [Thu Oct 17 16:43:37 2019]  do_exit+0x284/0xb20
> [Thu Oct 17 16:43:37 2019]  ? handle_mm_fault+0xfa/0x200
> [Thu Oct 17 16:43:37 2019]  do_group_exit+0x3a/0xa0
> [Thu Oct 17 16:43:37 2019]  __x64_sys_exit_group+0x14/0x20
> [Thu Oct 17 16:43:37 2019]  do_syscall_64+0x4f/0x100
> [Thu Oct 17 16:43:37 2019]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> Suggested-by: Felix Kuehling 
> Signed-off-by: Philip Yang 
> ---
>   drivers/gpu/drm/amd/amdkfd/kfd_chardev.c|  6 +++---
>   drivers/gpu/drm/amd/amdkfd/kfd_device.c |  5 -
>   .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.c   | 13 ++---
>   .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.h   |  1 +
>   4 files changed, 14 insertions(+), 11 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> index d9e36dbf13d5..40d75c39f08e 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> @@ -120,13 +120,13 @@ static int kfd_open(struct inode *inode, struct file 
> *filep)
>   return -EPERM;
>   }
>   
> + if (kfd_is_locked())
> + return -EAGAIN;
> +
>   process = kfd_create_process(filep);
>   if (IS_ERR(process))
>   return PTR_ERR(process);
>   
> - if (kfd_is_locked())
> - return -EAGAIN;
> -
>   dev_dbg(kfd_device, "process %d opened, compat mode (32 bit) - %d\n",
>   process->pasid, process->is_32bit_user_mode);
>   
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
> index 8f4b24e84964..4fa8834ce7cb 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
> @@ -730,9 +730,6 @@ int kgd2kfd_pre_reset(struct kfd_dev *kfd)
>   return 0;
>   kgd2kfd_suspend(kfd);
>   
> - /* hold dqm->lock to prevent further execution*/
> - dqm_lock(kfd->dqm);
> -
>   kfd_signal_reset_event(kfd);
>   return 0;
>   }
> @@ -750,8 +747,6 @@ int kgd2kfd_post_reset(struct kfd_dev *kfd)
>   if (!kfd->init_complete)
>   return 0;
>   
> - dqm_unlock(kfd->dqm);
> -
>   ret = kfd_resume(kfd)

Re: [PATCH] drm/amdkfd: don't use dqm lock during device reset/suspend/resume

[Kuehling, Felix]

On 2019-10-21 22:02, Zeng, Oak wrote:
> If we decline the queue creation request in suspend state by returning 
> -EAGAIN, then this approach works for both hws and non-hws. This way the 
> driver is clean but application need to re-create queue later when it get a 
> EAGAIN. Currently application is not aware of the suspend/resume state, so it 
> is hard for application to know when to re-create queue.
>
> The main benefit to allowing create queue in suspend state is that it is 
> easier for application writer. But no actual performance gain as no task will 
> be executed in suspend state.

We should not need to prevent queue creation while suspended. The 
processes are suspended. That means new queues will be created in 
evicted state:

     /*
  * Eviction state logic: mark all queues as evicted, even ones
  * not currently active. Restoring inactive queues later only
  * updates the is_evicted flag but is a no-op otherwise.
  */
     q->properties.is_evicted = !!qpd->evicted;

mqd_mgr->load_mqd will only be called for active queues. So even in the 
non-HWS case we should not be touching the HW while suspended. But I'd 
like to see some safeguards in place to make sure those assumptions are 
never violated.

Regards,
   Felix


>
> Regards,
> Oak
>
> -Original Message-
> From: amd-gfx  On Behalf Of Kuehling, 
> Felix
> Sent: Monday, October 21, 2019 9:04 PM
> To: Yang, Philip ; amd-gfx@lists.freedesktop.org
> Subject: Re: [PATCH] drm/amdkfd: don't use dqm lock during device 
> reset/suspend/resume
>
>
> On 2019-10-21 5:04 p.m., Yang, Philip wrote:
>> If device reset/suspend/resume failed for some reason, dqm lock is
>> hold forever and this causes deadlock. Below is a kernel backtrace
>> when application open kfd after suspend/resume failed.
>>
>> Instead of holding dqm lock in pre_reset and releasing dqm lock in
>> post_reset, add dqm->device_stopped flag which is modified in
>> dqm->ops.start and dqm->ops.stop. The flag doesn't need lock
>> dqm->protection
>> because write/read are all inside dqm lock.
>>
>> For HWS case, map_queues_cpsch and unmap_queues_cpsch checks
>> device_stopped flag before sending the updated runlist.
> What about the non-HWS case?
>
> In theory in non-HWS case new queues should be created in evicted state while 
> the device (and all processes) are suspended. So we should never try to map 
> or unmap queues to HQDs during suspend. But I'd feel better with a WARN_ON 
> and error return in the right places to make sure we're not missing anything. 
> Basically, we can't call any of the load/destroy_mqd functions while 
> suspended.
>
> That reminds me, we also have to add some checks in the debugfs code to avoid 
> dumping HQDs of a DQM that's stopped.
>
> Last comment: dqm->device_stopped must be initialized as true. It will get 
> set to false when the device is first started. It may be easier to reverse 
> the logic, something like dqm->sched_running that gets implicitly initialized 
> as false.
>
> Regards,
>     Felix
>
>
>> Backtrace of dqm lock deadlock:
>>
>> [Thu Oct 17 16:43:37 2019] INFO: task rocminfo:3024 blocked for more
>> than 120 seconds.
>> [Thu Oct 17 16:43:37 2019]   Not tainted
>> 5.0.0-rc1-kfd-compute-rocm-dkms-no-npi-1131 #1 [Thu Oct 17 16:43:37
>> 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>> message.
>> [Thu Oct 17 16:43:37 2019] rocminfoD0  3024   2947
>> 0x8000
>> [Thu Oct 17 16:43:37 2019] Call Trace:
>> [Thu Oct 17 16:43:37 2019]  ? __schedule+0x3d9/0x8a0 [Thu Oct 17
>> 16:43:37 2019]  schedule+0x32/0x70 [Thu Oct 17 16:43:37 2019]
>> schedule_preempt_disabled+0xa/0x10
>> [Thu Oct 17 16:43:37 2019]  __mutex_lock.isra.9+0x1e3/0x4e0 [Thu Oct
>> 17 16:43:37 2019]  ? __call_srcu+0x264/0x3b0 [Thu Oct 17 16:43:37
>> 2019]  ? process_termination_cpsch+0x24/0x2f0
>> [amdgpu]
>> [Thu Oct 17 16:43:37 2019]  process_termination_cpsch+0x24/0x2f0
>> [amdgpu]
>> [Thu Oct 17 16:43:37 2019]
>> kfd_process_dequeue_from_all_devices+0x42/0x60 [amdgpu] [Thu Oct 17
>> 16:43:37 2019]  kfd_process_notifier_release+0x1be/0x220
>> [amdgpu]
>> [Thu Oct 17 16:43:37 2019]  __mmu_notifier_release+0x3e/0xc0 [Thu Oct
>> 17 16:43:37 2019]  exit_mmap+0x160/0x1a0 [Thu Oct 17 16:43:37 2019]  ?
>> __handle_mm_fault+0xba3/0x1200 [Thu Oct 17 16:43:37 2019]  ?
>> exit_robust_list+0x5a/0x110 [Thu Oct 17 16:43:37 2019]
>> mmput+0x4a/0x120 [Thu Oct 17 16:43:37 2019]  do_exit+0x284/0xb20 [Thu
>> Oct 17 16:43:37 2019]  ? handle_mm_fault+

Re: [PATCH v2] drm/amdkfd: don't use dqm lock during device reset/suspend/resume

[Kuehling, Felix]

On 2019-10-22 14:28, Yang, Philip wrote:
> If device reset/suspend/resume failed for some reason, dqm lock is
> hold forever and this causes deadlock. Below is a kernel backtrace when
> application open kfd after suspend/resume failed.
>
> Instead of holding dqm lock in pre_reset and releasing dqm lock in
> post_reset, add dqm->device_stopped flag which is modified in
> dqm->ops.start and dqm->ops.stop. The flag doesn't need lock protection
> because write/read are all inside dqm lock.
>
> For HWS case, map_queues_cpsch and unmap_queues_cpsch checks
> device_stopped flag before sending the updated runlist.
>
> v2: For no-HWS case, when device is stopped, don't call
> load/destroy_mqd for eviction, restore and create queue, and avoid
> debugfs dump hdqs.
>
> Backtrace of dqm lock deadlock:
>
> [Thu Oct 17 16:43:37 2019] INFO: task rocminfo:3024 blocked for more
> than 120 seconds.
> [Thu Oct 17 16:43:37 2019]   Not tainted
> 5.0.0-rc1-kfd-compute-rocm-dkms-no-npi-1131 #1
> [Thu Oct 17 16:43:37 2019] "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [Thu Oct 17 16:43:37 2019] rocminfoD0  3024   2947
> 0x8000
> [Thu Oct 17 16:43:37 2019] Call Trace:
> [Thu Oct 17 16:43:37 2019]  ? __schedule+0x3d9/0x8a0
> [Thu Oct 17 16:43:37 2019]  schedule+0x32/0x70
> [Thu Oct 17 16:43:37 2019]  schedule_preempt_disabled+0xa/0x10
> [Thu Oct 17 16:43:37 2019]  __mutex_lock.isra.9+0x1e3/0x4e0
> [Thu Oct 17 16:43:37 2019]  ? __call_srcu+0x264/0x3b0
> [Thu Oct 17 16:43:37 2019]  ? process_termination_cpsch+0x24/0x2f0
> [amdgpu]
> [Thu Oct 17 16:43:37 2019]  process_termination_cpsch+0x24/0x2f0
> [amdgpu]
> [Thu Oct 17 16:43:37 2019]
> kfd_process_dequeue_from_all_devices+0x42/0x60 [amdgpu]
> [Thu Oct 17 16:43:37 2019]  kfd_process_notifier_release+0x1be/0x220
> [amdgpu]
> [Thu Oct 17 16:43:37 2019]  __mmu_notifier_release+0x3e/0xc0
> [Thu Oct 17 16:43:37 2019]  exit_mmap+0x160/0x1a0
> [Thu Oct 17 16:43:37 2019]  ? __handle_mm_fault+0xba3/0x1200
> [Thu Oct 17 16:43:37 2019]  ? exit_robust_list+0x5a/0x110
> [Thu Oct 17 16:43:37 2019]  mmput+0x4a/0x120
> [Thu Oct 17 16:43:37 2019]  do_exit+0x284/0xb20
> [Thu Oct 17 16:43:37 2019]  ? handle_mm_fault+0xfa/0x200
> [Thu Oct 17 16:43:37 2019]  do_group_exit+0x3a/0xa0
> [Thu Oct 17 16:43:37 2019]  __x64_sys_exit_group+0x14/0x20
> [Thu Oct 17 16:43:37 2019]  do_syscall_64+0x4f/0x100
> [Thu Oct 17 16:43:37 2019]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> Suggested-by: Felix Kuehling 
> Signed-off-by: Philip Yang 

Three more comments inline. With those comments addressed, this patch is

Reviewed-by: Felix Kuehling 


> ---
>   drivers/gpu/drm/amd/amdkfd/kfd_chardev.c  |  6 +--
>   drivers/gpu/drm/amd/amdkfd/kfd_device.c   |  5 --
>   .../drm/amd/amdkfd/kfd_device_queue_manager.c | 47 +--
>   .../drm/amd/amdkfd/kfd_device_queue_manager.h |  1 +
>   4 files changed, 46 insertions(+), 13 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> index d9e36dbf13d5..40d75c39f08e 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> @@ -120,13 +120,13 @@ static int kfd_open(struct inode *inode, struct file 
> *filep)
>   return -EPERM;
>   }
>   
> + if (kfd_is_locked())
> + return -EAGAIN;
> +
>   process = kfd_create_process(filep);
>   if (IS_ERR(process))
>   return PTR_ERR(process);
>   
> - if (kfd_is_locked())
> - return -EAGAIN;
> -

Is this part of the change still needed? I remember that this sequence 
was a bit tricky with some potential race condition when Shaoyun was 
working on it. This may have unintended side effects.


>   dev_dbg(kfd_device, "process %d opened, compat mode (32 bit) - %d\n",
>   process->pasid, process->is_32bit_user_mode);
>   
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
> index 8f4b24e84964..4fa8834ce7cb 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
> @@ -730,9 +730,6 @@ int kgd2kfd_pre_reset(struct kfd_dev *kfd)
>   return 0;
>   kgd2kfd_suspend(kfd);
>   
> - /* hold dqm->lock to prevent further execution*/
> - dqm_lock(kfd->dqm);
> -
>   kfd_signal_reset_event(kfd);
>   return 0;
>   }
> @@ -750,8 +747,6 @@ int kgd2kfd_post_reset(struct kfd_dev *kfd)
>   if (!kfd->init_complete)
>   return 0;
>   
> - dqm_unlock(kfd->dqm);
> -
>   ret = kfd_resume(kfd);
>   if (ret)
>   return ret;
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
> index 81fb545cf42c..82e1c6280d13 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
> @@ -340,6 +340,10 @@ static int cre

Re: [PATCH] drm/amdkfd: bug fix for out of bounds mem on gpu cache filling info

[Kuehling, Felix]

On 2019-10-24 14:46, Sierra Guiza, Alejandro (Alex) wrote:
> The bitmap in cu_info structure is defined as a 4x4 size array. In
> Acturus, this matrix is initialized as a 4x2. Based on the 8 shaders.
> In the gpu cache filling initialization, the access to the bitmap matrix
> was done as an 8x1 instead of 4x2. Causing an out of bounds memory
> access error.
> Due to this, the number of GPU cache entries was inconsistent.
>
> Signed-off-by: Alex Sierra 

Reviewed-by: Felix Kuehling 


> ---
>   drivers/gpu/drm/amd/amdkfd/kfd_crat.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
> index 0c327e0fc0f7..de9f68d5c312 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
> @@ -710,7 +710,7 @@ static int kfd_fill_gpu_cache_info(struct kfd_dev *kdev,
>   pcache_info,
>   cu_info,
>   mem_available,
> - cu_info->cu_bitmap[i][j],
> + cu_info->cu_bitmap[i % 4][j + i 
> / 4],
>   ct,
>   cu_processor_id,
>   k);
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdkfd: Delete duplicated queue bit map reservation

[Kuehling, Felix]

On 2019-10-24 5:14 p.m., Zhao, Yong wrote:
> The KIQ is on the second MEC and its reservation is covered in the
> latter logic, so no need to reserve its bit twice.
>
> Change-Id: Ieee390953a60c7d43de5a9aec38803f1f583a4a9
> Signed-off-by: Yong Zhao 

Reviewed-by: Felix Kuehling 


> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 8 
>   1 file changed, 8 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
> index 8c531793fe17..d3da9dde4ee1 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
> @@ -130,14 +130,6 @@ void amdgpu_amdkfd_device_init(struct amdgpu_device 
> *adev)
> adev->gfx.mec.queue_bitmap,
> KGD_MAX_QUEUES);
>   
> - /* remove the KIQ bit as well */
> - if (adev->gfx.kiq.ring.sched.ready)
> - clear_bit(amdgpu_gfx_mec_queue_to_bit(adev,
> -   adev->gfx.kiq.ring.me 
> - 1,
> -   
> adev->gfx.kiq.ring.pipe,
> -   
> adev->gfx.kiq.ring.queue),
> -   gpu_resources.queue_bitmap);
> -
>   /* According to linux/bitmap.h we shouldn't use bitmap_clear if
>* nbits is not compile time constant
>*/
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH v2 12/15] drm/amdgpu: Call find_vma under mmap_sem

[Kuehling, Felix]

On 2019-10-28 4:10 p.m., Jason Gunthorpe wrote:
> From: Jason Gunthorpe 
>
> find_vma() must be called under the mmap_sem, reorganize this code to
> do the vma check after entering the lock.
>
> Further, fix the unlocked use of struct task_struct's mm, instead use
> the mm from hmm_mirror which has an active mm_grab. Also the mm_grab
> must be converted to a mm_get before acquiring mmap_sem or calling
> find_vma().
>
> Fixes: 66c45500bfdc ("drm/amdgpu: use new HMM APIs and helpers")
> Fixes: 0919195f2b0d ("drm/amdgpu: Enable amdgpu_ttm_tt_get_user_pages in 
> worker threads")
> Cc: Alex Deucher 
> Cc: Christian König 
> Cc: David (ChunMing) Zhou 
> Cc: amd-gfx@lists.freedesktop.org
> Signed-off-by: Jason Gunthorpe 

One question inline to confirm my understanding. Otherwise this patch is

Reviewed-by: Felix Kuehling 


> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 37 ++---
>   1 file changed, 21 insertions(+), 16 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> index dff41d0a85fe96..c0e41f1f0c2365 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> @@ -35,6 +35,7 @@
>   #include 
>   #include 
>   #include 
> +#include 
>   #include 
>   #include 
>   #include 
> @@ -788,7 +789,7 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, 
> struct page **pages)
>   struct hmm_mirror *mirror = bo->mn ? &bo->mn->mirror : NULL;
>   struct ttm_tt *ttm = bo->tbo.ttm;
>   struct amdgpu_ttm_tt *gtt = (void *)ttm;
> - struct mm_struct *mm = gtt->usertask->mm;
> + struct mm_struct *mm;
>   unsigned long start = gtt->userptr;
>   struct vm_area_struct *vma;
>   struct hmm_range *range;
> @@ -796,25 +797,14 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, 
> struct page **pages)
>   uint64_t *pfns;
>   int r = 0;
>   
> - if (!mm) /* Happens during process shutdown */
> - return -ESRCH;
> -
>   if (unlikely(!mirror)) {
>   DRM_DEBUG_DRIVER("Failed to get hmm_mirror\n");
> - r = -EFAULT;
> - goto out;
> + return -EFAULT;
>   }
>   
> - vma = find_vma(mm, start);
> - if (unlikely(!vma || start < vma->vm_start)) {
> - r = -EFAULT;
> - goto out;
> - }
> - if (unlikely((gtt->userflags & AMDGPU_GEM_USERPTR_ANONONLY) &&
> - vma->vm_file)) {
> - r = -EPERM;
> - goto out;
> - }
> + mm = mirror->hmm->mmu_notifier.mm;
> + if (!mmget_not_zero(mm)) /* Happens during process shutdown */

This works because mirror->hmm->mmu_notifier holds an mmgrab reference 
to the mm? So the MM will not just go away, but if the mmget refcount is 
0, it means the mm is marked for destruction and shouldn't be used any more.


> + return -ESRCH;
>   
>   range = kzalloc(sizeof(*range), GFP_KERNEL);
>   if (unlikely(!range)) {
> @@ -847,6 +837,17 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, 
> struct page **pages)
>   hmm_range_wait_until_valid(range, HMM_RANGE_DEFAULT_TIMEOUT);
>   
>   down_read(&mm->mmap_sem);
> + vma = find_vma(mm, start);
> + if (unlikely(!vma || start < vma->vm_start)) {
> + r = -EFAULT;
> + goto out_unlock;
> + }
> + if (unlikely((gtt->userflags & AMDGPU_GEM_USERPTR_ANONONLY) &&
> + vma->vm_file)) {
> + r = -EPERM;
> + goto out_unlock;
> + }
> +
>   r = hmm_range_fault(range, 0);
>   up_read(&mm->mmap_sem);
>   
> @@ -865,15 +866,19 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, 
> struct page **pages)
>   }
>   
>   gtt->range = range;
> + mmput(mm);
>   
>   return 0;
>   
> +out_unlock:
> + up_read(&mm->mmap_sem);
>   out_free_pfns:
>   hmm_range_unregister(range);
>   kvfree(pfns);
>   out_free_ranges:
>   kfree(range);
>   out:
> + mmput(mm);
>   return r;
>   }
>   
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH v2 02/15] mm/mmu_notifier: add an interval tree notifier

[Kuehling, Felix]

I haven't had enough time to fully understand the deferred logic in this 
change. I spotted one problem, see comments inline.

On 2019-10-28 4:10 p.m., Jason Gunthorpe wrote:
> From: Jason Gunthorpe 
>
> Of the 13 users of mmu_notifiers, 8 of them use only
> invalidate_range_start/end() and immediately intersect the
> mmu_notifier_range with some kind of internal list of VAs.  4 use an
> interval tree (i915_gem, radeon_mn, umem_odp, hfi1). 4 use a linked list
> of some kind (scif_dma, vhost, gntdev, hmm)
>
> And the remaining 5 either don't use invalidate_range_start() or do some
> special thing with it.
>
> It turns out that building a correct scheme with an interval tree is
> pretty complicated, particularly if the use case is synchronizing against
> another thread doing get_user_pages().  Many of these implementations have
> various subtle and difficult to fix races.
>
> This approach puts the interval tree as common code at the top of the mmu
> notifier call tree and implements a shareable locking scheme.
>
> It includes:
>   - An interval tree tracking VA ranges, with per-range callbacks
>   - A read/write locking scheme for the interval tree that avoids
> sleeping in the notifier path (for OOM killer)
>   - A sequence counter based collision-retry locking scheme to tell
> device page fault that a VA range is being concurrently invalidated.
>
> This is based on various ideas:
> - hmm accumulates invalidated VA ranges and releases them when all
>invalidates are done, via active_invalidate_ranges count.
>This approach avoids having to intersect the interval tree twice (as
>umem_odp does) at the potential cost of a longer device page fault.
>
> - kvm/umem_odp use a sequence counter to drive the collision retry,
>via invalidate_seq
>
> - a deferred work todo list on unlock scheme like RTNL, via deferred_list.
>This makes adding/removing interval tree members more deterministic
>
> - seqlock, except this version makes the seqlock idea multi-holder on the
>write side by protecting it with active_invalidate_ranges and a spinlock
>
> To minimize MM overhead when only the interval tree is being used, the
> entire SRCU and hlist overheads are dropped using some simple
> branches. Similarly the interval tree overhead is dropped when in hlist
> mode.
>
> The overhead from the mandatory spinlock is broadly the same as most of
> existing users which already had a lock (or two) of some sort on the
> invalidation path.
>
> Cc: Andrea Arcangeli 
> Cc: Michal Hocko 
> Acked-by: Christian König 
> Signed-off-by: Jason Gunthorpe 
> ---
>   include/linux/mmu_notifier.h |  98 +++
>   mm/Kconfig   |   1 +
>   mm/mmu_notifier.c| 533 +--
>   3 files changed, 607 insertions(+), 25 deletions(-)
>
[snip]
> diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c
> index 367670cfd02b7b..d02d3c8c223eb7 100644
> --- a/mm/mmu_notifier.c
> +++ b/mm/mmu_notifier.c
[snip]
>* because mm->mm_users > 0 during mmu_notifier_register and exit_mmap
> @@ -52,17 +286,24 @@ struct mmu_notifier_mm {
>* can't go away from under us as exit_mmap holds an mm_count pin
>* itself.
>*/
> -void __mmu_notifier_release(struct mm_struct *mm)
> +static void mn_hlist_release(struct mmu_notifier_mm *mmn_mm,
> +  struct mm_struct *mm)
>   {
>   struct mmu_notifier *mn;
>   int id;
>   
> + if (mmn_mm->has_interval)
> + mn_itree_release(mmn_mm, mm);
> +
> + if (hlist_empty(&mmn_mm->list))
> + return;

This seems to duplicate the conditions in __mmu_notifier_release. See my 
comments below, I think one of them is wrong. I suspect this one, 
because __mmu_notifier_release follows the same pattern as the other 
notifiers.


> +
>   /*
>* SRCU here will block mmu_notifier_unregister until
>* ->release returns.
>*/
>   id = srcu_read_lock(&srcu);
> - hlist_for_each_entry_rcu(mn, &mm->mmu_notifier_mm->list, hlist)
> + hlist_for_each_entry_rcu(mn, &mmn_mm->list, hlist)
>   /*
>* If ->release runs before mmu_notifier_unregister it must be
>* handled, as it's the only way for the driver to flush all
> @@ -72,9 +313,9 @@ void __mmu_notifier_release(struct mm_struct *mm)
>   if (mn->ops->release)
>   mn->ops->release(mn, mm);
>   
> - spin_lock(&mm->mmu_notifier_mm->lock);
> - while (unlikely(!hlist_empty(&mm->mmu_notifier_mm->list))) {
> - mn = hlist_entry(mm->mmu_notifier_mm->list.first,
> + spin_lock(&mmn_mm->lock);
> + while (unlikely(!hlist_empty(&mmn_mm->list))) {
> + mn = hlist_entry(mmn_mm->list.first,
>struct mmu_notifier,
>hlist);
>   /*
> @@ -85,7 +326,7 @@ void __mmu_notifier_release(struct mm_struct *mm)
>*/
>   hlist_del_init_rcu(&mn->hl

Re: [PATCH v2 13/15] drm/amdgpu: Use mmu_range_insert instead of hmm_mirror

[Kuehling, Felix]

On 2019-10-28 4:10 p.m., Jason Gunthorpe wrote:
> From: Jason Gunthorpe 
>
> Remove the interval tree in the driver and rely on the tree maintained by
> the mmu_notifier for delivering mmu_notifier invalidation callbacks.
>
> For some reason amdgpu has a very complicated arrangement where it tries
> to prevent duplicate entries in the interval_tree, this is not necessary,
> each amdgpu_bo can be its own stand alone entry. interval_tree already
> allows duplicates and overlaps in the tree.
>
> Also, there is no need to remove entries upon a release callback, the
> mmu_range API safely allows objects to remain registered beyond the
> lifetime of the mm. The driver only has to stop touching the pages during
> release.
>
> Cc: Alex Deucher 
> Cc: Christian König 
> Cc: David (ChunMing) Zhou 
> Cc: amd-gfx@lists.freedesktop.org
> Signed-off-by: Jason Gunthorpe 
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu.h   |   2 +
>   .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |   5 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c|   1 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c| 341 --
>   drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h|   4 -
>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.h|  13 +-
>   6 files changed, 84 insertions(+), 282 deletions(-)
[snip]
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c
> index 31d4deb5d29484..4ffd7b90f4d907 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c
[snip]
> @@ -50,66 +50,6 @@
>   #include "amdgpu.h"
>   #include "amdgpu_amdkfd.h"
>   
> -/**
> - * struct amdgpu_mn_node
> - *
> - * @it: interval node defining start-last of the affected address range
> - * @bos: list of all BOs in the affected address range
> - *
> - * Manages all BOs which are affected of a certain range of address space.
> - */
> -struct amdgpu_mn_node {
> - struct interval_tree_node   it;
> - struct list_headbos;
> -};
> -
> -/**
> - * amdgpu_mn_destroy - destroy the HMM mirror
> - *
> - * @work: previously sheduled work item
> - *
> - * Lazy destroys the notifier from a work item
> - */
> -static void amdgpu_mn_destroy(struct work_struct *work)
> -{
> - struct amdgpu_mn *amn = container_of(work, struct amdgpu_mn, work);
> - struct amdgpu_device *adev = amn->adev;
> - struct amdgpu_mn_node *node, *next_node;
> - struct amdgpu_bo *bo, *next_bo;
> -
> - mutex_lock(&adev->mn_lock);
> - down_write(&amn->lock);
> - hash_del(&amn->node);
> - rbtree_postorder_for_each_entry_safe(node, next_node,
> -  &amn->objects.rb_root, it.rb) {
> - list_for_each_entry_safe(bo, next_bo, &node->bos, mn_list) {
> - bo->mn = NULL;
> - list_del_init(&bo->mn_list);
> - }
> - kfree(node);
> - }
> - up_write(&amn->lock);
> - mutex_unlock(&adev->mn_lock);
> -
> - hmm_mirror_unregister(&amn->mirror);
> - kfree(amn);
> -}
> -
> -/**
> - * amdgpu_hmm_mirror_release - callback to notify about mm destruction
> - *
> - * @mirror: the HMM mirror (mm) this callback is about
> - *
> - * Shedule a work item to lazy destroy HMM mirror.
> - */
> -static void amdgpu_hmm_mirror_release(struct hmm_mirror *mirror)
> -{
> - struct amdgpu_mn *amn = container_of(mirror, struct amdgpu_mn, mirror);
> -
> - INIT_WORK(&amn->work, amdgpu_mn_destroy);
> - schedule_work(&amn->work);
> -}
> -
>   /**
>* amdgpu_mn_lock - take the write side lock for this notifier
>*
> @@ -133,157 +73,86 @@ void amdgpu_mn_unlock(struct amdgpu_mn *mn)
>   }
>   
>   /**
> - * amdgpu_mn_read_lock - take the read side lock for this notifier
> - *
> - * @amn: our notifier
> - */
> -static int amdgpu_mn_read_lock(struct amdgpu_mn *amn, bool blockable)
> -{
> - if (blockable)
> - down_read(&amn->lock);
> - else if (!down_read_trylock(&amn->lock))
> - return -EAGAIN;
> -
> - return 0;
> -}
> -
> -/**
> - * amdgpu_mn_read_unlock - drop the read side lock for this notifier
> - *
> - * @amn: our notifier
> - */
> -static void amdgpu_mn_read_unlock(struct amdgpu_mn *amn)
> -{
> - up_read(&amn->lock);
> -}
> -
> -/**
> - * amdgpu_mn_invalidate_node - unmap all BOs of a node
> + * amdgpu_mn_invalidate_gfx - callback to notify about mm change
>*
> - * @node: the node with the BOs to unmap
> - * @start: start of address range affected
> - * @end: end of address range affected
> + * @mrn: the range (mm) is about to update
> + * @range: details on the invalidation
>*
>* Block for operations on BOs to finish and mark pages as accessed and
>* potentially dirty.
>*/
> -static void amdgpu_mn_invalidate_node(struct amdgpu_mn_node *node,
> -   unsigned long start,
> -   unsigned long end)
> +static bool amdgpu_mn_invalidate_gfx(stru

Re: [PATCH] drm/amdgpu: remove PT BOs when unmapping

[Kuehling, Felix]

On 2019-10-30 9:52 a.m., Christian König wrote:
> Am 29.10.19 um 21:06 schrieb Huang, JinHuiEric:
>> The issue is PT BOs are not freed when unmapping VA,
>> which causes vram usage accumulated is huge in some
>> memory stress test, such as kfd big buffer stress test.
>> Function amdgpu_vm_bo_update_mapping() is called by both
>> amdgpu_vm_bo_update() and amdgpu_vm_clear_freed(). The
>> solution is replacing amdgpu_vm_bo_update_mapping() in
>> amdgpu_vm_clear_freed() with removing PT BOs function
>> to save vram usage.
>
> NAK, that is intentional behavior.
>
> Otherwise we can run into out of memory situations when page tables 
> need to be allocated again under stress.

That's a bit arbitrary and inconsistent. We are freeing page tables in 
other situations, when a mapping uses huge pages in 
amdgpu_vm_update_ptes. Why not when a mapping is destroyed completely?

I'm actually a bit surprised that the huge-page handling in 
amdgpu_vm_update_ptes isn't kicking in to free up lower-level page 
tables when a BO is unmapped.

Regards,
   Felix


>
> Regards,
> Christian.
>
>>
>> Change-Id: Ic24e35bff8ca85265b418a642373f189d972a924
>> Signed-off-by: Eric Huang 
>> ---
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 56 
>> +-
>>   1 file changed, 48 insertions(+), 8 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> index 0f4c3b2..8a480c7 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>> @@ -1930,6 +1930,51 @@ static void amdgpu_vm_prt_fini(struct 
>> amdgpu_device *adev, struct amdgpu_vm *vm)
>>   }
>>     /**
>> + * amdgpu_vm_remove_ptes - free PT BOs
>> + *
>> + * @adev: amdgpu device structure
>> + * @vm: amdgpu vm structure
>> + * @start: start of mapped range
>> + * @end: end of mapped entry
>> + *
>> + * Free the page table level.
>> + */
>> +static int amdgpu_vm_remove_ptes(struct amdgpu_device *adev,
>> +    struct amdgpu_vm *vm, uint64_t start, uint64_t end)
>> +{
>> +    struct amdgpu_vm_pt_cursor cursor;
>> +    unsigned shift, num_entries;
>> +
>> +    amdgpu_vm_pt_start(adev, vm, start, &cursor);
>> +    while (cursor.level < AMDGPU_VM_PTB) {
>> +    if (!amdgpu_vm_pt_descendant(adev, &cursor))
>> +    return -ENOENT;
>> +    }
>> +
>> +    while (cursor.pfn < end) {
>> +    amdgpu_vm_free_table(cursor.entry);
>> +    num_entries = amdgpu_vm_num_entries(adev, cursor.level - 1);
>> +
>> +    if (cursor.entry != &cursor.parent->entries[num_entries - 1]) {
>> +    /* Next ptb entry */
>> +    shift = amdgpu_vm_level_shift(adev, cursor.level - 1);
>> +    cursor.pfn += 1ULL << shift;
>> +    cursor.pfn &= ~((1ULL << shift) - 1);
>> +    cursor.entry++;
>> +    } else {
>> +    /* Next ptb entry in next pd0 entry */
>> +    amdgpu_vm_pt_ancestor(&cursor);
>> +    shift = amdgpu_vm_level_shift(adev, cursor.level - 1);
>> +    cursor.pfn += 1ULL << shift;
>> +    cursor.pfn &= ~((1ULL << shift) - 1);
>> +    amdgpu_vm_pt_descendant(adev, &cursor);
>> +    }
>> +    }
>> +
>> +    return 0;
>> +}
>> +
>> +/**
>>    * amdgpu_vm_clear_freed - clear freed BOs in the PT
>>    *
>>    * @adev: amdgpu_device pointer
>> @@ -1949,7 +1994,6 @@ int amdgpu_vm_clear_freed(struct amdgpu_device 
>> *adev,
>>     struct dma_fence **fence)
>>   {
>>   struct amdgpu_bo_va_mapping *mapping;
>> -    uint64_t init_pte_value = 0;
>>   struct dma_fence *f = NULL;
>>   int r;
>>   @@ -1958,13 +2002,10 @@ int amdgpu_vm_clear_freed(struct 
>> amdgpu_device *adev,
>>   struct amdgpu_bo_va_mapping, list);
>>   list_del(&mapping->list);
>>   -    if (vm->pte_support_ats &&
>> -    mapping->start < AMDGPU_GMC_HOLE_START)
>> -    init_pte_value = AMDGPU_PTE_DEFAULT_ATC;
>> +    r = amdgpu_vm_remove_ptes(adev, vm,
>> +    (mapping->start + 0x1ff) & (~0x1ffll),
>> +    (mapping->last + 1) & (~0x1ffll));
>>   -    r = amdgpu_vm_bo_update_mapping(adev, vm, false, NULL,
>> -    mapping->start, mapping->last,
>> -    init_pte_value, 0, NULL, &f);
>>   amdgpu_vm_free_mapping(adev, vm, mapping, f);
>>   if (r) {
>>   dma_fence_put(f);
>> @@ -1980,7 +2021,6 @@ int amdgpu_vm_clear_freed(struct amdgpu_device 
>> *adev,
>>   }
>>     return 0;
>> -
>>   }
>>     /**
>
> ___
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdkfd: Simplify the mmap offset related bit operations

[Kuehling, Felix]

NAK. This won't work for several reasons.

The mmap_offset is used as offset parameter in the mmap system call. If 
you check the man page of mmap, you'll see that "offset must be a 
multiple of the page size". Therefore the PAGE_SHIFT is necessary.

In the case of doorbell offsets, the offset is parsed and processed by 
the Thunk in user mode. On GFX9 GPUs the lower bits are used for the 
offset of the doorbell within the doorbell page. On GFX8 the queue ID 
was used, but on GFX9 we had to decoupled the doorbell ID from the queue 
ID. If you remove the PAGE_SHIFT, you'll need to put those bits 
somewhere else. But that change in the encoding would break the ABI with 
the Thunk.

Regards,
   Felix

On 2019-11-01 4:48 p.m., Zhao, Yong wrote:
> The new code is much cleaner and results in better readability.
>
> Change-Id: I0c1f7cca7e24ddb7b4ffe1cb0fa71943828ae373
> Signed-off-by: Yong Zhao 
> ---
>   drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 13 +++--
>   drivers/gpu/drm/amd/amdkfd/kfd_events.c  |  1 -
>   drivers/gpu/drm/amd/amdkfd/kfd_priv.h|  9 +++--
>   drivers/gpu/drm/amd/amdkfd/kfd_process.c |  3 +--
>   4 files changed, 11 insertions(+), 15 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> index b91993753b82..590138727ca9 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> @@ -298,7 +298,6 @@ static int kfd_ioctl_create_queue(struct file *filep, 
> struct kfd_process *p,
>   /* Return gpu_id as doorbell offset for mmap usage */
>   args->doorbell_offset = KFD_MMAP_TYPE_DOORBELL;
>   args->doorbell_offset |= KFD_MMAP_GPU_ID(args->gpu_id);
> - args->doorbell_offset <<= PAGE_SHIFT;
>   if (KFD_IS_SOC15(dev->device_info->asic_family))
>   /* On SOC15 ASICs, include the doorbell offset within the
>* process doorbell frame, which could be 1 page or 2 pages.
> @@ -1938,20 +1937,22 @@ static int kfd_mmap(struct file *filp, struct 
> vm_area_struct *vma)
>   {
>   struct kfd_process *process;
>   struct kfd_dev *dev = NULL;
> - unsigned long vm_pgoff;
> + unsigned long mmap_offset;
>   unsigned int gpu_id;
>   
>   process = kfd_get_process(current);
>   if (IS_ERR(process))
>   return PTR_ERR(process);
>   
> - vm_pgoff = vma->vm_pgoff;
> - vma->vm_pgoff = KFD_MMAP_OFFSET_VALUE_GET(vm_pgoff);
> - gpu_id = KFD_MMAP_GPU_ID_GET(vm_pgoff);
> + mmap_offset = vma->vm_pgoff << PAGE_SHIFT;
> + gpu_id = KFD_MMAP_GET_GPU_ID(mmap_offset);
>   if (gpu_id)
>   dev = kfd_device_by_id(gpu_id);
>   
> - switch (vm_pgoff & KFD_MMAP_TYPE_MASK) {
> + /* only leave the offset segment */
> + vma->vm_pgoff &= (1ULL << (KFD_MMAP_GPU_ID_SHIFT - PAGE_SHIFT)) - 1;
> +
> + switch (mmap_offset & KFD_MMAP_TYPE_MASK) {
>   case KFD_MMAP_TYPE_DOORBELL:
>   if (!dev)
>   return -ENODEV;
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_events.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_events.c
> index 908081c85de1..1f8365575b12 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_events.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_events.c
> @@ -346,7 +346,6 @@ int kfd_event_create(struct file *devkfd, struct 
> kfd_process *p,
>   ret = create_signal_event(devkfd, p, ev);
>   if (!ret) {
>   *event_page_offset = KFD_MMAP_TYPE_EVENTS;
> - *event_page_offset <<= PAGE_SHIFT;
>   *event_slot_index = ev->event_id;
>   }
>   break;
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h 
> b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
> index 66bae8f2dad1..8eecd2cd1fd2 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
> @@ -59,24 +59,21 @@
>* NOTE: struct vm_area_struct.vm_pgoff uses offset in pages. Hence, these
>*  defines are w.r.t to PAGE_SIZE
>*/
> -#define KFD_MMAP_TYPE_SHIFT  (62 - PAGE_SHIFT)
> +#define KFD_MMAP_TYPE_SHIFT  (62)
>   #define KFD_MMAP_TYPE_MASK  (0x3ULL << KFD_MMAP_TYPE_SHIFT)
>   #define KFD_MMAP_TYPE_DOORBELL  (0x3ULL << KFD_MMAP_TYPE_SHIFT)
>   #define KFD_MMAP_TYPE_EVENTS(0x2ULL << KFD_MMAP_TYPE_SHIFT)
>   #define KFD_MMAP_TYPE_RESERVED_MEM  (0x1ULL << KFD_MMAP_TYPE_SHIFT)
>   #define KFD_MMAP_TYPE_MMIO  (0x0ULL << KFD_MMAP_TYPE_SHIFT)
>   
> -#define KFD_MMAP_GPU_ID_SHIFT (46 - PAGE_SHIFT)
> +#define KFD_MMAP_GPU_ID_SHIFT (46)
>   #define KFD_MMAP_GPU_ID_MASK (((1ULL << KFD_GPU_ID_HASH_WIDTH) - 1) \
>   << KFD_MMAP_GPU_ID_SHIFT)
>   #define KFD_MMAP_GPU_ID(gpu_id) uint64_t)gpu_id) << 
> KFD_MMAP_GPU_ID_SHIFT)\
>   & KFD_MMAP_GPU_ID_MASK)
> -#define KFD_MMAP_GPU_ID_GET(offset)((offset & KFD_MMAP_GPU_ID_MASK) \
> +#define KFD_MMAP_GET_GPU_ID(offset)((offset & KFD_MMAP_GPU_ID_MASK) \
>  

Re: [PATCH] drm/amdkfd: Simplify the mmap offset related bit operations

[Kuehling, Felix]

On 2019-11-01 4:48 p.m., Zhao, Yong wrote:
> The new code is much cleaner and results in better readability.
>
> Change-Id: I0c1f7cca7e24ddb7b4ffe1cb0fa71943828ae373
> Signed-off-by: Yong Zhao 
> ---
>   drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 13 +++--
>   drivers/gpu/drm/amd/amdkfd/kfd_events.c  |  1 -
>   drivers/gpu/drm/amd/amdkfd/kfd_priv.h|  9 +++--
>   drivers/gpu/drm/amd/amdkfd/kfd_process.c |  3 +--
>   4 files changed, 11 insertions(+), 15 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> index b91993753b82..590138727ca9 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> @@ -298,7 +298,6 @@ static int kfd_ioctl_create_queue(struct file *filep, 
> struct kfd_process *p,
>   /* Return gpu_id as doorbell offset for mmap usage */
>   args->doorbell_offset = KFD_MMAP_TYPE_DOORBELL;
>   args->doorbell_offset |= KFD_MMAP_GPU_ID(args->gpu_id);
> - args->doorbell_offset <<= PAGE_SHIFT;
>   if (KFD_IS_SOC15(dev->device_info->asic_family))
>   /* On SOC15 ASICs, include the doorbell offset within the
>* process doorbell frame, which could be 1 page or 2 pages.
> @@ -1938,20 +1937,22 @@ static int kfd_mmap(struct file *filp, struct 
> vm_area_struct *vma)
>   {
>   struct kfd_process *process;
>   struct kfd_dev *dev = NULL;
> - unsigned long vm_pgoff;
> + unsigned long mmap_offset;
>   unsigned int gpu_id;
>   
>   process = kfd_get_process(current);
>   if (IS_ERR(process))
>   return PTR_ERR(process);
>   
> - vm_pgoff = vma->vm_pgoff;
> - vma->vm_pgoff = KFD_MMAP_OFFSET_VALUE_GET(vm_pgoff);
> - gpu_id = KFD_MMAP_GPU_ID_GET(vm_pgoff);
> + mmap_offset = vma->vm_pgoff << PAGE_SHIFT;
> + gpu_id = KFD_MMAP_GET_GPU_ID(mmap_offset);
>   if (gpu_id)
>   dev = kfd_device_by_id(gpu_id);
>   
> - switch (vm_pgoff & KFD_MMAP_TYPE_MASK) {
> + /* only leave the offset segment */
> + vma->vm_pgoff &= (1ULL << (KFD_MMAP_GPU_ID_SHIFT - PAGE_SHIFT)) - 1;

You're now open-coding what used to be done by the 
KFD_MMAP_OFFSET_VALUE_GET macro. I don't see how this is an improvement. 
Maybe better to update the macro to do this.


> +
> + switch (mmap_offset & KFD_MMAP_TYPE_MASK) {
>   case KFD_MMAP_TYPE_DOORBELL:
>   if (!dev)
>   return -ENODEV;
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_events.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_events.c
> index 908081c85de1..1f8365575b12 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_events.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_events.c
> @@ -346,7 +346,6 @@ int kfd_event_create(struct file *devkfd, struct 
> kfd_process *p,
>   ret = create_signal_event(devkfd, p, ev);
>   if (!ret) {
>   *event_page_offset = KFD_MMAP_TYPE_EVENTS;
> - *event_page_offset <<= PAGE_SHIFT;
>   *event_slot_index = ev->event_id;
>   }
>   break;
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h 
> b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
> index 66bae8f2dad1..8eecd2cd1fd2 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
> @@ -59,24 +59,21 @@
>* NOTE: struct vm_area_struct.vm_pgoff uses offset in pages. Hence, these
>*  defines are w.r.t to PAGE_SIZE
>*/
> -#define KFD_MMAP_TYPE_SHIFT  (62 - PAGE_SHIFT)
> +#define KFD_MMAP_TYPE_SHIFT  (62)
>   #define KFD_MMAP_TYPE_MASK  (0x3ULL << KFD_MMAP_TYPE_SHIFT)
>   #define KFD_MMAP_TYPE_DOORBELL  (0x3ULL << KFD_MMAP_TYPE_SHIFT)
>   #define KFD_MMAP_TYPE_EVENTS(0x2ULL << KFD_MMAP_TYPE_SHIFT)
>   #define KFD_MMAP_TYPE_RESERVED_MEM  (0x1ULL << KFD_MMAP_TYPE_SHIFT)
>   #define KFD_MMAP_TYPE_MMIO  (0x0ULL << KFD_MMAP_TYPE_SHIFT)
>   
> -#define KFD_MMAP_GPU_ID_SHIFT (46 - PAGE_SHIFT)
> +#define KFD_MMAP_GPU_ID_SHIFT (46)
>   #define KFD_MMAP_GPU_ID_MASK (((1ULL << KFD_GPU_ID_HASH_WIDTH) - 1) \
>   << KFD_MMAP_GPU_ID_SHIFT)
>   #define KFD_MMAP_GPU_ID(gpu_id) uint64_t)gpu_id) << 
> KFD_MMAP_GPU_ID_SHIFT)\
>   & KFD_MMAP_GPU_ID_MASK)
> -#define KFD_MMAP_GPU_ID_GET(offset)((offset & KFD_MMAP_GPU_ID_MASK) \
> +#define KFD_MMAP_GET_GPU_ID(offset)((offset & KFD_MMAP_GPU_ID_MASK) \
>   >> KFD_MMAP_GPU_ID_SHIFT)
>   
> -#define KFD_MMAP_OFFSET_VALUE_MASK   (0x3FFFULL >> PAGE_SHIFT)
> -#define KFD_MMAP_OFFSET_VALUE_GET(offset) (offset & 
> KFD_MMAP_OFFSET_VALUE_MASK)

This macro is still useful. See above. I think you should just update 
the mask and continue using it for clarity.

Regards,
   Felix


> -
>   /*
>* When working with cp scheduler we should assign the HIQ manually or via
>* the amdgpu driver to a fixed hqd slot, here are the fixed HIQ hqd slot
> diff --git a/dr

Re: [PATCH] drm/amdgpu: change read of GPU clock counter on Vega10 VF

[Kuehling, Felix]

On 2019-11-05 5:03 p.m., Huang, JinHuiEric wrote:
> Using unified VBIOS has performance drop in sriov environment.
> The fix is switching to another register instead.
>
> Signed-off-by: Eric Huang 
> ---
>   drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 18 +++---
>   1 file changed, 15 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c 
> b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
> index 829d623..6770bd1 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
> @@ -3885,9 +3885,21 @@ static uint64_t gfx_v9_0_get_gpu_clock_counter(struct 
> amdgpu_device *adev)
>   uint64_t clock;
>   
>   mutex_lock(&adev->gfx.gpu_clock_mutex);
> - WREG32_SOC15(GC, 0, mmRLC_CAPTURE_GPU_CLOCK_COUNT, 1);
> - clock = (uint64_t)RREG32_SOC15(GC, 0, mmRLC_GPU_CLOCK_COUNT_LSB) |
> - ((uint64_t)RREG32_SOC15(GC, 0, mmRLC_GPU_CLOCK_COUNT_MSB) << 
> 32ULL);
> + if (adev->asic_type == CHIP_VEGA10 && amdgpu_sriov_runtime(adev)) {
> + uint32_t tmp, lsb, msb, i = 0;
> + do {
> + tmp = RREG32_SOC15(GC, 0, mmRLC_REFCLOCK_TIMESTAMP_MSB);
> + lsb = RREG32_SOC15(GC, 0, mmRLC_REFCLOCK_TIMESTAMP_LSB);
> + msb = RREG32_SOC15(GC, 0, mmRLC_REFCLOCK_TIMESTAMP_MSB);
> + i++;
> + udelay(1);

This udelay should be conditional. In the likely case that tmp == msb, 
you should never have to delay at all. Maybe put the delay at the start 
of the loop with a condition:

     if (i != 0)
         udelay(1);

So that it only applies before the second and later loops.

Regards,
   Felix


> + } while (unlikely(tmp != msb) && (i < adev->usec_timeout));
> + clock = (uint64_t)lsb | ((uint64_t)msb << 32ULL);
> + } else {
> + WREG32_SOC15(GC, 0, mmRLC_CAPTURE_GPU_CLOCK_COUNT, 1);
> + clock = (uint64_t)RREG32_SOC15(GC, 0, 
> mmRLC_GPU_CLOCK_COUNT_LSB) |
> + ((uint64_t)RREG32_SOC15(GC, 0, 
> mmRLC_GPU_CLOCK_COUNT_MSB) << 32ULL);
> + }
>   mutex_unlock(&adev->gfx.gpu_clock_mutex);
>   return clock;
>   }
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdkfd: Simplify the mmap offset related bit operations

[Kuehling, Felix]

On 2019-11-01 7:03 p.m., Zhao, Yong wrote:
>
> > + /* only leave the offset segment */
> > + vma->vm_pgoff &= (1ULL << (KFD_MMAP_GPU_ID_SHIFT - PAGE_SHIFT)) - 
> 1;
>
> You're now open-coding what used to be done by the
> KFD_MMAP_OFFSET_VALUE_GET macro. I don't see how this is an
> improvement.
> Maybe better to update the macro to do this.
>
>
> I can definitely do that, but I think we'd better delete this line 
> completely as it seems odd to change vm_pgoff. Moreover this vm_pgoff 
> is not used at all in the following function calls. What do you think?

I think you're right. Looks like a historical accident. I see that older 
versions of kfd_event_mmap used to access vm_pgoff and probably depended 
on this. We removed that in this commit:


commit 50cb7dd94cb43a6204813376e1be1d21780b71fb
Author: Felix Kuehling 
Date:   Fri Oct 27 19:35:26 2017 -0400

     drm/amdkfd: Simplify events page allocator

     The first event page is always big enough to handle all events.
     Handling of multiple events pages is not supported by user mode, and
     not necessary.

     Signed-off-by: Yong Zhao 
     Signed-off-by: Felix Kuehling 
     Acked-by: Oded Gabbay 
     Signed-off-by: Oded Gabbay 



Regards,
   Felix


>
> Regards,
> Yong
> ----
> *From:* Kuehling, Felix 
> *Sent:* Friday, November 1, 2019 6:48 PM
> *To:* Zhao, Yong ; amd-gfx@lists.freedesktop.org 
> 
> *Subject:* Re: [PATCH] drm/amdkfd: Simplify the mmap offset related 
> bit operations
> On 2019-11-01 4:48 p.m., Zhao, Yong wrote:
> > The new code is much cleaner and results in better readability.
> >
> > Change-Id: I0c1f7cca7e24ddb7b4ffe1cb0fa71943828ae373
> > Signed-off-by: Yong Zhao 
> > ---
> >   drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 13 +++--
> >   drivers/gpu/drm/amd/amdkfd/kfd_events.c  |  1 -
> >   drivers/gpu/drm/amd/amdkfd/kfd_priv.h    |  9 +++--
> >   drivers/gpu/drm/amd/amdkfd/kfd_process.c |  3 +--
> >   4 files changed, 11 insertions(+), 15 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> > index b91993753b82..590138727ca9 100644
> > --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> > +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> > @@ -298,7 +298,6 @@ static int kfd_ioctl_create_queue(struct file 
> *filep, struct kfd_process *p,
> >    /* Return gpu_id as doorbell offset for mmap usage */
> >    args->doorbell_offset = KFD_MMAP_TYPE_DOORBELL;
> >    args->doorbell_offset |= KFD_MMAP_GPU_ID(args->gpu_id);
> > - args->doorbell_offset <<= PAGE_SHIFT;
> >    if (KFD_IS_SOC15(dev->device_info->asic_family))
> >    /* On SOC15 ASICs, include the doorbell offset within the
> > * process doorbell frame, which could be 1 page or 2 
> pages.
> > @@ -1938,20 +1937,22 @@ static int kfd_mmap(struct file *filp, 
> struct vm_area_struct *vma)
> >   {
> >    struct kfd_process *process;
> >    struct kfd_dev *dev = NULL;
> > - unsigned long vm_pgoff;
> > + unsigned long mmap_offset;
> >    unsigned int gpu_id;
> >
> >    process = kfd_get_process(current);
> >    if (IS_ERR(process))
> >    return PTR_ERR(process);
> >
> > - vm_pgoff = vma->vm_pgoff;
> > - vma->vm_pgoff = KFD_MMAP_OFFSET_VALUE_GET(vm_pgoff);
> > - gpu_id = KFD_MMAP_GPU_ID_GET(vm_pgoff);
> > + mmap_offset = vma->vm_pgoff << PAGE_SHIFT;
> > + gpu_id = KFD_MMAP_GET_GPU_ID(mmap_offset);
> >    if (gpu_id)
> >    dev = kfd_device_by_id(gpu_id);
> >
> > - switch (vm_pgoff & KFD_MMAP_TYPE_MASK) {
> > + /* only leave the offset segment */
> > + vma->vm_pgoff &= (1ULL << (KFD_MMAP_GPU_ID_SHIFT - 
> PAGE_SHIFT)) - 1;
>
> You're now open-coding what used to be done by the
> KFD_MMAP_OFFSET_VALUE_GET macro. I don't see how this is an improvement.
> Maybe better to update the macro to do this.
>
>
> > +
> > + switch (mmap_offset & KFD_MMAP_TYPE_MASK) {
> >    case KFD_MMAP_TYPE_DOORBELL:
> >    if (!dev)
> >    return -ENODEV;
> > diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_events.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_events.c
> > index 908081c85de1..1f8365575b12 100644
> > --- a/drivers/gpu/drm/amd/amdkfd/kfd_events.c
> > +++ b/drivers/gpu/drm/amd/amdkfd/kfd_events.c
> > @@ -

Re: [PATCH] drm/amdgpu: change read of GPU clock counter on Vega10 VF

[Kuehling, Felix]

On 2019-11-05 5:26 p.m., Huang, JinHuiEric wrote:
> Using unified VBIOS has performance drop in sriov environment.
> The fix is switching to another register instead.
>
> Signed-off-by: Eric Huang 

Reviewed-by: Felix Kuehling 


> ---
>   drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 19 ---
>   1 file changed, 16 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c 
> b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
> index 829d623..e44a3ea 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
> @@ -3885,9 +3885,22 @@ static uint64_t gfx_v9_0_get_gpu_clock_counter(struct 
> amdgpu_device *adev)
>   uint64_t clock;
>   
>   mutex_lock(&adev->gfx.gpu_clock_mutex);
> - WREG32_SOC15(GC, 0, mmRLC_CAPTURE_GPU_CLOCK_COUNT, 1);
> - clock = (uint64_t)RREG32_SOC15(GC, 0, mmRLC_GPU_CLOCK_COUNT_LSB) |
> - ((uint64_t)RREG32_SOC15(GC, 0, mmRLC_GPU_CLOCK_COUNT_MSB) << 
> 32ULL);
> + if (adev->asic_type == CHIP_VEGA10 && amdgpu_sriov_runtime(adev)) {
> + uint32_t tmp, lsb, msb, i = 0;
> + do {
> + if (i != 0)
> + udelay(1);
> + tmp = RREG32_SOC15(GC, 0, mmRLC_REFCLOCK_TIMESTAMP_MSB);
> + lsb = RREG32_SOC15(GC, 0, mmRLC_REFCLOCK_TIMESTAMP_LSB);
> + msb = RREG32_SOC15(GC, 0, mmRLC_REFCLOCK_TIMESTAMP_MSB);
> + i++;
> + } while (unlikely(tmp != msb) && (i < adev->usec_timeout));
> + clock = (uint64_t)lsb | ((uint64_t)msb << 32ULL);
> + } else {
> + WREG32_SOC15(GC, 0, mmRLC_CAPTURE_GPU_CLOCK_COUNT, 1);
> + clock = (uint64_t)RREG32_SOC15(GC, 0, 
> mmRLC_GPU_CLOCK_COUNT_LSB) |
> + ((uint64_t)RREG32_SOC15(GC, 0, 
> mmRLC_GPU_CLOCK_COUNT_MSB) << 32ULL);
> + }
>   mutex_unlock(&adev->gfx.gpu_clock_mutex);
>   return clock;
>   }
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 2/3] drm/amdkfd: only keep release_mem function for Hawaii

[Kuehling, Felix]

On 2019-10-30 20:17, Zhao, Yong wrote:
> release_mem won't be used at all on GFX9 and GFX10, so delete it.

Hawaii was GFXv7. So we're not using the release_mem packet on GFXv8 
either. Why arbitrarily limit this change to GFXv9 and 10?

Regards,
   Felix

>
> Change-Id: I13787a8a29b83e7516c582a7401f2e14721edf5f
> Signed-off-by: Yong Zhao 
> ---
>   .../gpu/drm/amd/amdkfd/kfd_kernel_queue_v10.c | 35 ++-
>   .../gpu/drm/amd/amdkfd/kfd_kernel_queue_v9.c  | 33 ++---
>   2 files changed, 4 insertions(+), 64 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v10.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v10.c
> index aed32ab7102e..bfd6221acae9 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v10.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v10.c
> @@ -298,37 +298,6 @@ static int pm_query_status_v10(struct packet_manager 
> *pm, uint32_t *buffer,
>   return 0;
>   }
>   
> -
> -static int pm_release_mem_v10(uint64_t gpu_addr, uint32_t *buffer)
> -{
> - struct pm4_mec_release_mem *packet;
> -
> - WARN_ON(!buffer);
> -
> - packet = (struct pm4_mec_release_mem *)buffer;
> - memset(buffer, 0, sizeof(struct pm4_mec_release_mem));
> -
> - packet->header.u32All = pm_build_pm4_header(IT_RELEASE_MEM,
> - sizeof(struct pm4_mec_release_mem));
> -
> - packet->bitfields2.event_type = CACHE_FLUSH_AND_INV_TS_EVENT;
> - packet->bitfields2.event_index = 
> event_index__mec_release_mem__end_of_pipe;
> - packet->bitfields2.tcl1_action_ena = 1;
> - packet->bitfields2.tc_action_ena = 1;
> - packet->bitfields2.cache_policy = cache_policy__mec_release_mem__lru;
> -
> - packet->bitfields3.data_sel = 
> data_sel__mec_release_mem__send_32_bit_low;
> - packet->bitfields3.int_sel =
> - int_sel__mec_release_mem__send_interrupt_after_write_confirm;
> -
> - packet->bitfields4.address_lo_32b = (gpu_addr & 0x) >> 2;
> - packet->address_hi = upper_32_bits(gpu_addr);
> -
> - packet->data_lo = 0;
> -
> - return sizeof(struct pm4_mec_release_mem) / sizeof(unsigned int);
> -}
> -
>   const struct packet_manager_funcs kfd_v10_pm_funcs = {
>   .map_process= pm_map_process_v10,
>   .runlist= pm_runlist_v10,
> @@ -336,13 +305,13 @@ const struct packet_manager_funcs kfd_v10_pm_funcs = {
>   .map_queues = pm_map_queues_v10,
>   .unmap_queues   = pm_unmap_queues_v10,
>   .query_status   = pm_query_status_v10,
> - .release_mem= pm_release_mem_v10,
> + .release_mem= NULL,
>   .map_process_size   = sizeof(struct pm4_mes_map_process),
>   .runlist_size   = sizeof(struct pm4_mes_runlist),
>   .set_resources_size = sizeof(struct pm4_mes_set_resources),
>   .map_queues_size= sizeof(struct pm4_mes_map_queues),
>   .unmap_queues_size  = sizeof(struct pm4_mes_unmap_queues),
>   .query_status_size  = sizeof(struct pm4_mes_query_status),
> - .release_mem_size   = sizeof(struct pm4_mec_release_mem)
> + .release_mem_size   = 0,
>   };
>   
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v9.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v9.c
> index 3b5ca2b1d7a6..f0e4910a8865 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v9.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v9.c
> @@ -336,35 +336,6 @@ static int pm_query_status_v9(struct packet_manager *pm, 
> uint32_t *buffer,
>   return 0;
>   }
>   
> -
> -static int pm_release_mem_v9(uint64_t gpu_addr, uint32_t *buffer)
> -{
> - struct pm4_mec_release_mem *packet;
> -
> - packet = (struct pm4_mec_release_mem *)buffer;
> - memset(buffer, 0, sizeof(struct pm4_mec_release_mem));
> -
> - packet->header.u32All = pm_build_pm4_header(IT_RELEASE_MEM,
> - sizeof(struct pm4_mec_release_mem));
> -
> - packet->bitfields2.event_type = CACHE_FLUSH_AND_INV_TS_EVENT;
> - packet->bitfields2.event_index = 
> event_index__mec_release_mem__end_of_pipe;
> - packet->bitfields2.tcl1_action_ena = 1;
> - packet->bitfields2.tc_action_ena = 1;
> - packet->bitfields2.cache_policy = cache_policy__mec_release_mem__lru;
> -
> - packet->bitfields3.data_sel = 
> data_sel__mec_release_mem__send_32_bit_low;
> - packet->bitfields3.int_sel =
> - int_sel__mec_release_mem__send_interrupt_after_write_confirm;
> -
> - packet->bitfields4.address_lo_32b = (gpu_addr & 0x) >> 2;
> - packet->address_hi = upper_32_bits(gpu_addr);
> -
> - packet->data_lo = 0;
> -
> - return 0;
> -}
> -
>   const struct packet_manager_funcs kfd_v9_pm_funcs = {
>   .map_process= pm_map_process_v9,
>   .runlist

Re: [PATCH 1/3] drm/amdkfd: Adjust function sequences to avoid unnecessary declarations

[Kuehling, Felix]

On 2019-10-30 20:17, Zhao, Yong wrote:
> This is cleaner.
>
> Change-Id: I8cdecad387d8c547a088c6050f77385ee1135be1
> Signed-off-by: Yong Zhao 
Reviewed-by: Felix Kuehling 
> ---
>   .../gpu/drm/amd/amdkfd/kfd_kernel_queue_v9.c  | 19 +++
>   1 file changed, 7 insertions(+), 12 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v9.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v9.c
> index 9a4bafb2e175..3b5ca2b1d7a6 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v9.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v9.c
> @@ -26,18 +26,6 @@
>   #include "kfd_pm4_headers_ai.h"
>   #include "kfd_pm4_opcodes.h"
>   
> -static bool initialize_v9(struct kernel_queue *kq, struct kfd_dev *dev,
> - enum kfd_queue_type type, unsigned int queue_size);
> -static void uninitialize_v9(struct kernel_queue *kq);
> -static void submit_packet_v9(struct kernel_queue *kq);
> -
> -void kernel_queue_init_v9(struct kernel_queue_ops *ops)
> -{
> - ops->initialize = initialize_v9;
> - ops->uninitialize = uninitialize_v9;
> - ops->submit_packet = submit_packet_v9;
> -}
> -
>   static bool initialize_v9(struct kernel_queue *kq, struct kfd_dev *dev,
>   enum kfd_queue_type type, unsigned int queue_size)
>   {
> @@ -67,6 +55,13 @@ static void submit_packet_v9(struct kernel_queue *kq)
>   kq->pending_wptr64);
>   }
>   
> +void kernel_queue_init_v9(struct kernel_queue_ops *ops)
> +{
> + ops->initialize = initialize_v9;
> + ops->uninitialize = uninitialize_v9;
> + ops->submit_packet = submit_packet_v9;
> +}
> +
>   static int pm_map_process_v9(struct packet_manager *pm,
>   uint32_t *buffer, struct qcm_process_device *qpd)
>   {
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 3/3] drm/amdkfd: Use kernel queue v9 functions for v10

[Kuehling, Felix]

On 2019-10-30 20:17, Zhao, Yong wrote:
> The kernel queue functions for v9 and v10 are the same except
> pm_map_process_v* which have small difference, so they should be reused.
> This eliminates the need of reapplying several patches which were
> applied on v9 but not on v10, such as bigger GWS and more than 2
> SDMA engine support which were introduced on Arcturus.

This looks reasonable in principle. See one suggestion inline to 
simplify it further.


>
> Change-Id: I2d385961e3c884db14e30b5afc98d0d9e4cb1802
> Signed-off-by: Yong Zhao 
> ---
>   drivers/gpu/drm/amd/amdkfd/Makefile   |   1 -
>   drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.c |   4 +-
>   drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.h |   1 -
>   .../gpu/drm/amd/amdkfd/kfd_kernel_queue_v10.c | 317 --
>   .../gpu/drm/amd/amdkfd/kfd_kernel_queue_v9.c  |  49 ++-
>   drivers/gpu/drm/amd/amdkfd/kfd_priv.h |   3 -
>   6 files changed, 44 insertions(+), 331 deletions(-)
>   delete mode 100644 drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v10.c
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/Makefile 
> b/drivers/gpu/drm/amd/amdkfd/Makefile
> index 48155060a57c..017a8b7156da 100644
> --- a/drivers/gpu/drm/amd/amdkfd/Makefile
> +++ b/drivers/gpu/drm/amd/amdkfd/Makefile
> @@ -41,7 +41,6 @@ AMDKFD_FILES:= $(AMDKFD_PATH)/kfd_module.o \
>   $(AMDKFD_PATH)/kfd_kernel_queue_cik.o \
>   $(AMDKFD_PATH)/kfd_kernel_queue_vi.o \
>   $(AMDKFD_PATH)/kfd_kernel_queue_v9.o \
> - $(AMDKFD_PATH)/kfd_kernel_queue_v10.o \
>   $(AMDKFD_PATH)/kfd_packet_manager.o \
>   $(AMDKFD_PATH)/kfd_process_queue_manager.o \
>   $(AMDKFD_PATH)/kfd_device_queue_manager.o \
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.c
> index 11d244891393..0d966408ea87 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.c
> @@ -332,12 +332,10 @@ struct kernel_queue *kernel_queue_init(struct kfd_dev 
> *dev,
>   case CHIP_RAVEN:
>   case CHIP_RENOIR:
>   case CHIP_ARCTURUS:
> - kernel_queue_init_v9(&kq->ops_asic_specific);
> - break;
>   case CHIP_NAVI10:
>   case CHIP_NAVI12:
>   case CHIP_NAVI14:
> - kernel_queue_init_v10(&kq->ops_asic_specific);
> + kernel_queue_init_v9(&kq->ops_asic_specific);
>   break;
>   default:
>   WARN(1, "Unexpected ASIC family %u",
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.h 
> b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.h
> index 365fc674fea4..a7116a939029 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.h
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.h
> @@ -102,6 +102,5 @@ struct kernel_queue {
>   void kernel_queue_init_cik(struct kernel_queue_ops *ops);
>   void kernel_queue_init_vi(struct kernel_queue_ops *ops);
>   void kernel_queue_init_v9(struct kernel_queue_ops *ops);
> -void kernel_queue_init_v10(struct kernel_queue_ops *ops);
>   
>   #endif /* KFD_KERNEL_QUEUE_H_ */
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v10.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v10.c
> deleted file mode 100644
> index bfd6221acae9..
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v10.c
> +++ /dev/null
[snip]
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v9.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v9.c
> index f0e4910a8865..d8f7343bfe71 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v9.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v9.c
> @@ -62,8 +62,9 @@ void kernel_queue_init_v9(struct kernel_queue_ops *ops)
>   ops->submit_packet = submit_packet_v9;
>   }
>   
> -static int pm_map_process_v9(struct packet_manager *pm,
> - uint32_t *buffer, struct qcm_process_device *qpd)
> +static int pm_map_process_v9_base(struct packet_manager *pm,
> + uint32_t *buffer, struct qcm_process_device *qpd,
> + unsigned int sq_shader_tba_hi_trap_en_shift)
>   {
>   struct pm4_mes_map_process *packet;
>   uint64_t vm_page_table_base_addr = qpd->page_table_base;
> @@ -85,10 +86,16 @@ static int pm_map_process_v9(struct packet_manager *pm,
>   
>   packet->sh_mem_config = qpd->sh_mem_config;
>   packet->sh_mem_bases = qpd->sh_mem_bases;
> - packet->sq_shader_tba_lo = lower_32_bits(qpd->tba_addr >> 8);
> - packet->sq_shader_tba_hi = upper_32_bits(qpd->tba_addr >> 8);
> - packet->sq_shader_tma_lo = lower_32_bits(qpd->tma_addr >> 8);
> - packet->sq_shader_tma_hi = upper_32_bits(qpd->tma_addr >> 8);
> + if (qpd->tba_addr) {
> + packet->sq_shader_tba_lo = lower_32_bits(qpd->tba_addr >> 8);
> + packet->sq_shader_tba_hi = upper_32_bits(qpd->tba_addr >> 8);
> + if (sq_shader_tba_hi_trap_en_shift) {
> + packet->sq_shader_tba_hi 

Re: [PATCH] drm/amdkfd: Simplify the mmap offset related bit operations

[Kuehling, Felix]

On 2019-11-05 18:18, Zhao, Yong wrote:
> The new code uses straightforward bit shifts and thus has better 
> readability.

You're missing the MMAP-related code for mmio remapping. In 
kfd_ioctl_alloc_memory_of_gpu:

     /* MMIO is mapped through kfd device
  * Generate a kfd mmap offset
  */
     if (flags & KFD_IOC_ALLOC_MEM_FLAGS_MMIO_REMAP) {
     args->mmap_offset = KFD_MMAP_TYPE_MMIO | 
KFD_MMAP_GPU_ID(args->gpu_id);
     args->mmap_offset <<= PAGE_SHIFT;
     }

Regards,
   Felix

>
> Change-Id: I0c1f7cca7e24ddb7b4ffe1cb0fa71943828ae373
> Signed-off-by: Yong Zhao 
> ---
> drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 10 --
> drivers/gpu/drm/amd/amdkfd/kfd_events.c | 1 -
> drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 9 +++--
> drivers/gpu/drm/amd/amdkfd/kfd_process.c | 3 +--
> 4 files changed, 8 insertions(+), 15 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> index b91993753b82..34078df36621 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> @@ -298,7 +298,6 @@ static int kfd_ioctl_create_queue(struct file 
> *filep, struct kfd_process *p,
> /* Return gpu_id as doorbell offset for mmap usage */
> args->doorbell_offset = KFD_MMAP_TYPE_DOORBELL;
> args->doorbell_offset |= KFD_MMAP_GPU_ID(args->gpu_id);
> - args->doorbell_offset <<= PAGE_SHIFT;
> if (KFD_IS_SOC15(dev->device_info->asic_family))
> /* On SOC15 ASICs, include the doorbell offset within the
> * process doorbell frame, which could be 1 page or 2 pages.
> @@ -1938,20 +1937,19 @@ static int kfd_mmap(struct file *filp, struct 
> vm_area_struct *vma)
> {
> struct kfd_process *process;
> struct kfd_dev *dev = NULL;
> - unsigned long vm_pgoff;
> + unsigned long mmap_offset;
> unsigned int gpu_id;
> process = kfd_get_process(current);
> if (IS_ERR(process))
> return PTR_ERR(process);
> - vm_pgoff = vma->vm_pgoff;
> - vma->vm_pgoff = KFD_MMAP_OFFSET_VALUE_GET(vm_pgoff);
> - gpu_id = KFD_MMAP_GPU_ID_GET(vm_pgoff);
> + mmap_offset = vma->vm_pgoff << PAGE_SHIFT;
> + gpu_id = KFD_MMAP_GET_GPU_ID(mmap_offset);
> if (gpu_id)
> dev = kfd_device_by_id(gpu_id);
> - switch (vm_pgoff & KFD_MMAP_TYPE_MASK) {
> + switch (mmap_offset & KFD_MMAP_TYPE_MASK) {
> case KFD_MMAP_TYPE_DOORBELL:
> if (!dev)
> return -ENODEV;
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_events.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_events.c
> index 908081c85de1..1f8365575b12 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_events.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_events.c
> @@ -346,7 +346,6 @@ int kfd_event_create(struct file *devkfd, struct 
> kfd_process *p,
> ret = create_signal_event(devkfd, p, ev);
> if (!ret) {
> *event_page_offset = KFD_MMAP_TYPE_EVENTS;
> - *event_page_offset <<= PAGE_SHIFT;
> *event_slot_index = ev->event_id;
> }
> break;
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h 
> b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
> index 66bae8f2dad1..8eecd2cd1fd2 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
> @@ -59,24 +59,21 @@
> * NOTE: struct vm_area_struct.vm_pgoff uses offset in pages. Hence, these
> * defines are w.r.t to PAGE_SIZE
> */
> -#define KFD_MMAP_TYPE_SHIFT (62 - PAGE_SHIFT)
> +#define KFD_MMAP_TYPE_SHIFT (62)
> #define KFD_MMAP_TYPE_MASK (0x3ULL << KFD_MMAP_TYPE_SHIFT)
> #define KFD_MMAP_TYPE_DOORBELL (0x3ULL << KFD_MMAP_TYPE_SHIFT)
> #define KFD_MMAP_TYPE_EVENTS (0x2ULL << KFD_MMAP_TYPE_SHIFT)
> #define KFD_MMAP_TYPE_RESERVED_MEM (0x1ULL << KFD_MMAP_TYPE_SHIFT)
> #define KFD_MMAP_TYPE_MMIO (0x0ULL << KFD_MMAP_TYPE_SHIFT)
> -#define KFD_MMAP_GPU_ID_SHIFT (46 - PAGE_SHIFT)
> +#define KFD_MMAP_GPU_ID_SHIFT (46)
> #define KFD_MMAP_GPU_ID_MASK (((1ULL << KFD_GPU_ID_HASH_WIDTH) - 1) \
> << KFD_MMAP_GPU_ID_SHIFT)
> #define KFD_MMAP_GPU_ID(gpu_id) uint64_t)gpu_id) << 
> KFD_MMAP_GPU_ID_SHIFT)\
> & KFD_MMAP_GPU_ID_MASK)
> -#define KFD_MMAP_GPU_ID_GET(offset) ((offset & KFD_MMAP_GPU_ID_MASK) \
> +#define KFD_MMAP_GET_GPU_ID(offset) ((offset & KFD_MMAP_GPU_ID_MASK) \
> >> KFD_MMAP_GPU_ID_SHIFT)
> -#define KFD_MMAP_OFFSET_VALUE_MASK (0x3FFFULL >> PAGE_SHIFT)
> -#define KFD_MMAP_OFFSET_VALUE_GET(offset) (offset & 
> KFD_MMAP_OFFSET_VALUE_MASK)
> -
> /*
> * When working with cp scheduler we should assign the HIQ manually or via
> * the amdgpu driver to a fixed hqd slot, here are the fixed HIQ hqd slot
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_process.c
> index 6abfb77ae540..39dc49b8fd85 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c
> @@ -554,8 +554,7 @@ static int kfd_process_init_cwsr_apu(struct 
> kfd_process *p, struct file *filep)
> if (!dev->cwsr_enabled || qpd->cwsr_kaddr || qpd->cwsr_base)
> continue;
> - offset = (KFD_MMAP_TYPE_RESERVED_MEM | KFD_MMAP_GPU_ID(dev->id))
> - << PAGE_SHIFT;
> + offset = KFD_MMAP

Re: [PATCH 2/3] drm/amdkfd: only keep release_mem function for Hawaii

[Kuehling, Felix]

No, please lets not add a new nomenclature for PM4 packet versions. GFX 
versions are agreed on between hardware, firmware, and software and it's 
generally understood what they mean. If we add a new PM4 packet versioning 
scheme on our own, then this will add a lot of confusion when talking to 
firmware teams.

You know, this would all be more straight forward if we accepted a little bit 
of code duplication and had packet writing functions per GFX version. You'll 
see this pattern a lot in the amdgpu driver where each IP version duplicates a 
bunch of code. In many cases you may be able to save a few lines of code by 
sharing functions between IP versions. But you'll add some confusion and burden 
on future maintenance.

This is the price we pay for micro-optimizing minor code duplication.

Regards,
  Felix

On 2019-11-07 12:39, Zhao, Yong wrote:

Hi Kent,

I can't agree more on this. Also, the same applies to the file names. 
Definitely we need to agree on the naming scheme before making it happen.

Yong

On 2019-11-07 12:33 p.m., Russell, Kent wrote:
I think that the versioning is getting a little confusing since we’re using the 
old GFX versions, but not really sticking to it due to the shareability of 
certain managers and shaders. Could we look into doing something like gen1 or 
gen2, or some other more ambiguous non-GFX-related versioning? Otherwise we’ll 
keep having these questions of “why is Hawaii GFX8”, “why is Arcturus GFX9”, 
etc. Then if things change, we just up the value concretely, instead of maybe 
doing a v11 if GFX11 changes things, and only GFX11 ASICs use those 
functions/variables.

Obviously not high-priority, but maybe something to consider as you continue to 
consolidate and remove duplicate code.

Kent

From: amd-gfx 
<mailto:amd-gfx-boun...@lists.freedesktop.org>
 On Behalf Of Zhao, Yong
Sent: Thursday, November 7, 2019 11:57 AM
To: Kuehling, Felix <mailto:felix.kuehl...@amd.com>; 
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
Subject: Re: [PATCH 2/3] drm/amdkfd: only keep release_mem function for Hawaii

Hi Felix,

That's because v8 and v7 share the same packet_manager_funcs. In this case, it 
is better to keep it as it is.

Regards,
Yong
____
From: Kuehling, Felix mailto:felix.kuehl...@amd.com>>
Sent: Wednesday, November 6, 2019 11:45 PM
To: Zhao, Yong mailto:yong.z...@amd.com>>; 
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> 
mailto:amd-gfx@lists.freedesktop.org>>
Subject: Re: [PATCH 2/3] drm/amdkfd: only keep release_mem function for Hawaii

On 2019-10-30 20:17, Zhao, Yong wrote:
> release_mem won't be used at all on GFX9 and GFX10, so delete it.

Hawaii was GFXv7. So we're not using the release_mem packet on GFXv8
either. Why arbitrarily limit this change to GFXv9 and 10?

Regards,
   Felix

>
> Change-Id: I13787a8a29b83e7516c582a7401f2e14721edf5f
> Signed-off-by: Yong Zhao mailto:yong.z...@amd.com>>
> ---
>   .../gpu/drm/amd/amdkfd/kfd_kernel_queue_v10.c | 35 ++-
>   .../gpu/drm/amd/amdkfd/kfd_kernel_queue_v9.c  | 33 ++---
>   2 files changed, 4 insertions(+), 64 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v10.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v10.c
> index aed32ab7102e..bfd6221acae9 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v10.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v10.c
> @@ -298,37 +298,6 @@ static int pm_query_status_v10(struct packet_manager 
> *pm, uint32_t *buffer,
>return 0;
>   }
>
> -
> -static int pm_release_mem_v10(uint64_t gpu_addr, uint32_t *buffer)
> -{
> - struct pm4_mec_release_mem *packet;
> -
> - WARN_ON(!buffer);
> -
> - packet = (struct pm4_mec_release_mem *)buffer;
> - memset(buffer, 0, sizeof(struct pm4_mec_release_mem));
> -
> - packet->header.u32All = pm_build_pm4_header(IT_RELEASE_MEM,
> - sizeof(struct pm4_mec_release_mem));
> -
> - packet->bitfields2.event_type = CACHE_FLUSH_AND_INV_TS_EVENT;
> - packet->bitfields2.event_index = 
> event_index__mec_release_mem__end_of_pipe;
> - packet->bitfields2.tcl1_action_ena = 1;
> - packet->bitfields2.tc_action_ena = 1;
> - packet->bitfields2.cache_policy = cache_policy__mec_release_mem__lru;
> -
> - packet->bitfields3.data_sel = 
> data_sel__mec_release_mem__send_32_bit_low;
> - packet->bitfields3.int_sel =
> - int_sel__mec_release_mem__send_interrupt_after_write_confirm;
> -
> - packet->bitfields4.address_lo_32b = (gpu_addr & 0x) >> 2;
> - packet->address_hi = upper_32_bits(gpu_addr);
> -
> - packet->data_lo = 0;
> -
> - 

Re: [PATCH] drm/amdkfd: Simplify the mmap offset related bit operations

[Kuehling, Felix]

On 2019-11-07 12:33, Zhao, Yong wrote:
> The new code uses straightforward bit shifts and thus has better readability.
>
> Change-Id: I0c1f7cca7e24ddb7b4ffe1cb0fa71943828ae373
> Signed-off-by: Yong Zhao 

Reviewed-by: Felix Kuehling 


> ---
>   drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 17 +++--
>   drivers/gpu/drm/amd/amdkfd/kfd_events.c  |  1 -
>   drivers/gpu/drm/amd/amdkfd/kfd_priv.h|  9 +++--
>   drivers/gpu/drm/amd/amdkfd/kfd_process.c |  3 +--
>   4 files changed, 11 insertions(+), 19 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> index b91993753b82..e59c229861e6 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
> @@ -298,7 +298,6 @@ static int kfd_ioctl_create_queue(struct file *filep, 
> struct kfd_process *p,
>   /* Return gpu_id as doorbell offset for mmap usage */
>   args->doorbell_offset = KFD_MMAP_TYPE_DOORBELL;
>   args->doorbell_offset |= KFD_MMAP_GPU_ID(args->gpu_id);
> - args->doorbell_offset <<= PAGE_SHIFT;
>   if (KFD_IS_SOC15(dev->device_info->asic_family))
>   /* On SOC15 ASICs, include the doorbell offset within the
>* process doorbell frame, which could be 1 page or 2 pages.
> @@ -1312,10 +1311,9 @@ static int kfd_ioctl_alloc_memory_of_gpu(struct file 
> *filep,
>   /* MMIO is mapped through kfd device
>* Generate a kfd mmap offset
>*/
> - if (flags & KFD_IOC_ALLOC_MEM_FLAGS_MMIO_REMAP) {
> - args->mmap_offset = KFD_MMAP_TYPE_MMIO | 
> KFD_MMAP_GPU_ID(args->gpu_id);
> - args->mmap_offset <<= PAGE_SHIFT;
> - }
> + if (flags & KFD_IOC_ALLOC_MEM_FLAGS_MMIO_REMAP)
> + args->mmap_offset = KFD_MMAP_TYPE_MMIO
> + | KFD_MMAP_GPU_ID(args->gpu_id);
>   
>   return 0;
>   
> @@ -1938,20 +1936,19 @@ static int kfd_mmap(struct file *filp, struct 
> vm_area_struct *vma)
>   {
>   struct kfd_process *process;
>   struct kfd_dev *dev = NULL;
> - unsigned long vm_pgoff;
> + unsigned long mmap_offset;
>   unsigned int gpu_id;
>   
>   process = kfd_get_process(current);
>   if (IS_ERR(process))
>   return PTR_ERR(process);
>   
> - vm_pgoff = vma->vm_pgoff;
> - vma->vm_pgoff = KFD_MMAP_OFFSET_VALUE_GET(vm_pgoff);
> - gpu_id = KFD_MMAP_GPU_ID_GET(vm_pgoff);
> + mmap_offset = vma->vm_pgoff << PAGE_SHIFT;
> + gpu_id = KFD_MMAP_GET_GPU_ID(mmap_offset);
>   if (gpu_id)
>   dev = kfd_device_by_id(gpu_id);
>   
> - switch (vm_pgoff & KFD_MMAP_TYPE_MASK) {
> + switch (mmap_offset & KFD_MMAP_TYPE_MASK) {
>   case KFD_MMAP_TYPE_DOORBELL:
>   if (!dev)
>   return -ENODEV;
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_events.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_events.c
> index 908081c85de1..1f8365575b12 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_events.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_events.c
> @@ -346,7 +346,6 @@ int kfd_event_create(struct file *devkfd, struct 
> kfd_process *p,
>   ret = create_signal_event(devkfd, p, ev);
>   if (!ret) {
>   *event_page_offset = KFD_MMAP_TYPE_EVENTS;
> - *event_page_offset <<= PAGE_SHIFT;
>   *event_slot_index = ev->event_id;
>   }
>   break;
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h 
> b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
> index 66bae8f2dad1..8eecd2cd1fd2 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
> @@ -59,24 +59,21 @@
>* NOTE: struct vm_area_struct.vm_pgoff uses offset in pages. Hence, these
>*  defines are w.r.t to PAGE_SIZE
>*/
> -#define KFD_MMAP_TYPE_SHIFT  (62 - PAGE_SHIFT)
> +#define KFD_MMAP_TYPE_SHIFT  (62)
>   #define KFD_MMAP_TYPE_MASK  (0x3ULL << KFD_MMAP_TYPE_SHIFT)
>   #define KFD_MMAP_TYPE_DOORBELL  (0x3ULL << KFD_MMAP_TYPE_SHIFT)
>   #define KFD_MMAP_TYPE_EVENTS(0x2ULL << KFD_MMAP_TYPE_SHIFT)
>   #define KFD_MMAP_TYPE_RESERVED_MEM  (0x1ULL << KFD_MMAP_TYPE_SHIFT)
>   #define KFD_MMAP_TYPE_MMIO  (0x0ULL << KFD_MMAP_TYPE_SHIFT)
>   
> -#define KFD_MMAP_GPU_ID_SHIFT (46 - PAGE_SHIFT)
> +#define KFD_MMAP_GPU_ID_SHIFT (46)
>   #define KFD_MMAP_GPU_ID_MASK (((1ULL << KFD_GPU_ID_HASH_WIDTH) - 1) \
>   << KFD_MMAP_GPU_ID_SHIFT)
>   #define KFD_MMAP_GPU_ID(gpu_id) uint64_t)gpu_id) << 
> KFD_MMAP_GPU_ID_SHIFT)\
>   & KFD_MMAP_GPU_ID_MASK)
> -#define KFD_MMAP_GPU_ID_GET(offset)((offset & KFD_MMAP_GPU_ID_MASK) \
> +#define KFD_MMAP_GET_GPU_ID(offset)((offset & KFD_MMAP_GPU_ID_MASK) \
>   >> KFD_MMAP_GPU_ID_SHIFT)
>   
> -#define KFD_MMAP_OFFSET_VALUE_MASK   (0x3FFFULL >> PAGE_SHIFT)
> -#define KFD_MMAP_OFFSET_VALUE_GET(offset) (offset & 

Re: [PATCH 2/3] drm/amdkfd: only keep release_mem function for Hawaii

[Kuehling, Felix]

On 2019-11-07 13:54, Zhao, Yong wrote:
Hi Kent,

This consolidation is a must, because we should not have duplicated it in the 
first place.

The code is working fine with the duplication. You disagree with duplicating 
the code in the first place. But that's just your opinion. It's not a must in 
any objective sense.


The kernel queue functions by design are generic. The reasson why GFX8 and GFX9 
are different is because GFX9 is SOC15 where packet formats and doorbell size 
changed.  On the other hand, kfd_kernel_queue_v7.c file is pretty much empty by 
reusing v8 functions, even though it is there. Furthermore, in my opinion 
kfd_kernel_queue_v7.c should be merged into v8 counterpart, From GFX9 onwards, 
packet formats should stay the same. For kernel queues, we should be able to 
differentiate it by pre SOC15 or not, and I have an impression that MEC 
firmware agrees to maintain the kernel queue interface stable across 
generations most of time.

OK, you're making assumptions about PM4 packets on future ASIC generations. 
It's true that the transition to SOC15 with 64-bit doorbells and 
read/write-pointers was particularly disruptive. Your assumption will hold 
until it gets broken by some other disruptive change.


For now, if you want clear naming, we could call the GFXv7/8 packet manager 
functions "pre-SOC15" or "legacy" and the GFXv9/10 and future functions 
"SOC15". This may work for a while. But I suspect at some point something is 
going to change and we'll need to create a new version for a newer ASIC 
generation. You already have a small taste of that with the different 
TBA-enable bit in the MAP_PROCESS packet in GFXv10.


Regards,

  Felix


Regards,
Yong

From: Alex Deucher <mailto:alexdeuc...@gmail.com>
Sent: Thursday, November 7, 2019 1:32 PM
To: Kuehling, Felix <mailto:felix.kuehl...@amd.com>
Cc: Zhao, Yong <mailto:yong.z...@amd.com>; Russell, Kent 
<mailto:kent.russ...@amd.com>; 
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> 
<mailto:amd-gfx@lists.freedesktop.org>
Subject: Re: [PATCH 2/3] drm/amdkfd: only keep release_mem function for Hawaii

On Thu, Nov 7, 2019 at 12:47 PM Kuehling, Felix 
<mailto:felix.kuehl...@amd.com> wrote:
>
> No, please lets not add a new nomenclature for PM4 packet versions. GFX 
> versions are agreed on between hardware, firmware, and software and it's 
> generally understood what they mean. If we add a new PM4 packet versioning 
> scheme on our own, then this will add a lot of confusion when talking to 
> firmware teams.
>
> You know, this would all be more straight forward if we accepted a little bit 
> of code duplication and had packet writing functions per GFX version. You'll 
> see this pattern a lot in the amdgpu driver where each IP version duplicates 
> a bunch of code. In many cases you may be able to save a few lines of code by 
> sharing functions between IP versions. But you'll add some confusion and 
> burden on future maintenance.
>
> This is the price we pay for micro-optimizing minor code duplication.

What we've tried to do in amdgpu is to break out shared code in to
common helpers that are not IP specific and use that in each IP module
(e.g., amdgpu_uvd.c amdgpu_gfx.c, etc.).  Sometimes we can use a
particular chunk of code across multiple generations.  E.g., the uvd
stuff is a good example.  We have shared generic uvd helpers that work
for most UVD IP versions, and then if we need an IP specific version,
we override that in the callbacks with a version specific one.  E.g.,
for the video decode engines we use the generic helpers for a number
of ring functions:

static const struct amdgpu_ring_funcs uvd_v7_0_ring_vm_funcs = {
...
.test_ring = uvd_v7_0_ring_test_ring,
.test_ib = amdgpu_uvd_ring_test_ib,
.insert_nop = uvd_v7_0_ring_insert_nop,
.pad_ib = amdgpu_ring_generic_pad_ib,
.begin_use = amdgpu_uvd_ring_begin_use,
.end_use = amdgpu_uvd_ring_end_use,
...
};

while we override more of them for the video encode engines:

static const struct amdgpu_ring_funcs uvd_v7_0_enc_ring_vm_funcs = {
...
.test_ring = uvd_v7_0_enc_ring_test_ring,
.test_ib = uvd_v7_0_enc_ring_test_ib,
.insert_nop = amdgpu_ring_insert_nop,
.insert_end = uvd_v7_0_enc_ring_insert_end,
.pad_ib = amdgpu_ring_generic_pad_ib,
.begin_use = amdgpu_uvd_ring_begin_use,
.end_use = amdgpu_uvd_ring_end_use,
...
};

But still maintain IP specific components.

Alex

>
> Regards,
>   Felix
>
> On 2019-11-07 12:39, Zhao, Yong wrote:
>
> Hi Kent,
>
> I can't agree more on this. Also, the same applies to the file names. 
> Definitely we need to agree on the naming scheme before making it happen.
>
> Yong
>
> On 2019-11-07 12:33 p.m., Russe

Re: [PATCH 2/3] drm/amdkfd: only keep release_mem function for Hawaii

[Kuehling, Felix]

On 2019-11-07 14:40, Zhao, Yong wrote:
Hi Felix,

The code working fine is true except that all not new features after this 
duplication are broken. If I want to make all GFX10 feature complete, I have to 
either manually adapt several duplications to the GFX10 file or do this 
consolidation. From this perspective and ease of my work, it is a must.

"A must" means there is no alternative. You already listed two alternatives 
yourself: "either manually adapt several duplications to the GFX10 file or do 
this consolidation."


In _your_ opinion, the consolidation means less work for _you_. That's _your_ 
point of view. The discussion in this code review pointed out other points of 
view. When you take all of them into account, you may reconsider what is less 
work overall, and what is easier to maintain.


I'm not opposing your change per-se. But I'd like you to consider the whole 
picture, including the consequences of any design decisions you're making and 
imposing on anyone working on this code in the future. In this cases I think 
it's a relatively minor issue and it may just come down to a matter of opinion 
that I don't feel terribly strongly about.


With that said, the change is

Reviewed-by: Felix Kuehling 
<mailto:felix.kuehl...@amd.com>


Regards,

  Felix


Regards,
Yong


From: Kuehling, Felix <mailto:felix.kuehl...@amd.com>
Sent: Thursday, November 7, 2019 2:12 PM
To: Zhao, Yong <mailto:yong.z...@amd.com>; Alex Deucher 
<mailto:alexdeuc...@gmail.com>
Cc: Russell, Kent <mailto:kent.russ...@amd.com>; 
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> 
<mailto:amd-gfx@lists.freedesktop.org>
Subject: Re: [PATCH 2/3] drm/amdkfd: only keep release_mem function for Hawaii

On 2019-11-07 13:54, Zhao, Yong wrote:
Hi Kent,

This consolidation is a must, because we should not have duplicated it in the 
first place.

The code is working fine with the duplication. You disagree with duplicating 
the code in the first place. But that's just your opinion. It's not a must in 
any objective sense.



The kernel queue functions by design are generic. The reasson why GFX8 and GFX9 
are different is because GFX9 is SOC15 where packet formats and doorbell size 
changed.  On the other hand, kfd_kernel_queue_v7.c file is pretty much empty by 
reusing v8 functions, even though it is there. Furthermore, in my opinion 
kfd_kernel_queue_v7.c should be merged into v8 counterpart, From GFX9 onwards, 
packet formats should stay the same. For kernel queues, we should be able to 
differentiate it by pre SOC15 or not, and I have an impression that MEC 
firmware agrees to maintain the kernel queue interface stable across 
generations most of time.

OK, you're making assumptions about PM4 packets on future ASIC generations. 
It's true that the transition to SOC15 with 64-bit doorbells and 
read/write-pointers was particularly disruptive. Your assumption will hold 
until it gets broken by some other disruptive change.


For now, if you want clear naming, we could call the GFXv7/8 packet manager 
functions "pre-SOC15" or "legacy" and the GFXv9/10 and future functions 
"SOC15". This may work for a while. But I suspect at some point something is 
going to change and we'll need to create a new version for a newer ASIC 
generation. You already have a small taste of that with the different 
TBA-enable bit in the MAP_PROCESS packet in GFXv10.


Regards,

  Felix


Regards,
Yong

From: Alex Deucher <mailto:alexdeuc...@gmail.com>
Sent: Thursday, November 7, 2019 1:32 PM
To: Kuehling, Felix <mailto:felix.kuehl...@amd.com>
Cc: Zhao, Yong <mailto:yong.z...@amd.com>; Russell, Kent 
<mailto:kent.russ...@amd.com>; 
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> 
<mailto:amd-gfx@lists.freedesktop.org>
Subject: Re: [PATCH 2/3] drm/amdkfd: only keep release_mem function for Hawaii

On Thu, Nov 7, 2019 at 12:47 PM Kuehling, Felix 
<mailto:felix.kuehl...@amd.com> wrote:
>
> No, please lets not add a new nomenclature for PM4 packet versions. GFX 
> versions are agreed on between hardware, firmware, and software and it's 
> generally understood what they mean. If we add a new PM4 packet versioning 
> scheme on our own, then this will add a lot of confusion when talking to 
> firmware teams.
>
> You know, this would all be more straight forward if we accepted a little bit 
> of code duplication and had packet writing functions per GFX version. You'll 
> see this pattern a lot in the amdgpu driver where each IP version duplicates 
> a bunch of code. In many cases you may be able to save a few lines of code by 
> sharing functions between IP versions. But you'll add some confusion and 
> burden on future ma

Re: [PATCH 2/3] drm/amdkfd: only keep release_mem function for Hawaii

[Kuehling, Felix]

On 2019-11-07 13:32, Alex Deucher wrote:
> On Thu, Nov 7, 2019 at 12:47 PM Kuehling, Felix  
> wrote:
>> No, please lets not add a new nomenclature for PM4 packet versions. GFX 
>> versions are agreed on between hardware, firmware, and software and it's 
>> generally understood what they mean. If we add a new PM4 packet versioning 
>> scheme on our own, then this will add a lot of confusion when talking to 
>> firmware teams.
>>
>> You know, this would all be more straight forward if we accepted a little 
>> bit of code duplication and had packet writing functions per GFX version. 
>> You'll see this pattern a lot in the amdgpu driver where each IP version 
>> duplicates a bunch of code. In many cases you may be able to save a few 
>> lines of code by sharing functions between IP versions. But you'll add some 
>> confusion and burden on future maintenance.
>>
>> This is the price we pay for micro-optimizing minor code duplication.
> What we've tried to do in amdgpu is to break out shared code in to
> common helpers that are not IP specific and use that in each IP module
> (e.g., amdgpu_uvd.c amdgpu_gfx.c, etc.).  Sometimes we can use a
> particular chunk of code across multiple generations.  E.g., the uvd
> stuff is a good example.  We have shared generic uvd helpers that work
> for most UVD IP versions, and then if we need an IP specific version,
> we override that in the callbacks with a version specific one.  E.g.,
> for the video decode engines we use the generic helpers for a number
> of ring functions:
>
> static const struct amdgpu_ring_funcs uvd_v7_0_ring_vm_funcs = {
> ...
>  .test_ring = uvd_v7_0_ring_test_ring,
>  .test_ib = amdgpu_uvd_ring_test_ib,
>  .insert_nop = uvd_v7_0_ring_insert_nop,
>  .pad_ib = amdgpu_ring_generic_pad_ib,
>  .begin_use = amdgpu_uvd_ring_begin_use,
>  .end_use = amdgpu_uvd_ring_end_use,
> ...
> };
>
> while we override more of them for the video encode engines:
>
> static const struct amdgpu_ring_funcs uvd_v7_0_enc_ring_vm_funcs = {
> ...
>  .test_ring = uvd_v7_0_enc_ring_test_ring,
>  .test_ib = uvd_v7_0_enc_ring_test_ib,
>  .insert_nop = amdgpu_ring_insert_nop,
>  .insert_end = uvd_v7_0_enc_ring_insert_end,
>  .pad_ib = amdgpu_ring_generic_pad_ib,
>  .begin_use = amdgpu_uvd_ring_begin_use,
>  .end_use = amdgpu_uvd_ring_end_use,
> ...
> };
>
> But still maintain IP specific components.

Thanks Alex. In this case the common code is in kfd_packet_manager.c and 
the IP-version-specific code that writes the actual PM4 packets is in 
the kernel_queue_v*.c files. Yong is trying to merge the PM4 packet 
writing code for v9 and v10 because the packet formats are essentially 
unchanged. It makes the naming conventions in the code a bit meaningless 
because v9 now really means "v9 and v10". Apparently there is precedent 
for this, as we already did the same thing with v7 and v8, which I 
forgot about in my initial code review.

Regards,
   Felix


>
> Alex
>
>> Regards,
>>Felix
>>
>> On 2019-11-07 12:39, Zhao, Yong wrote:
>>
>> Hi Kent,
>>
>> I can't agree more on this. Also, the same applies to the file names. 
>> Definitely we need to agree on the naming scheme before making it happen.
>>
>> Yong
>>
>> On 2019-11-07 12:33 p.m., Russell, Kent wrote:
>>
>> I think that the versioning is getting a little confusing since we’re using 
>> the old GFX versions, but not really sticking to it due to the shareability 
>> of certain managers and shaders. Could we look into doing something like 
>> gen1 or gen2, or some other more ambiguous non-GFX-related versioning? 
>> Otherwise we’ll keep having these questions of “why is Hawaii GFX8”, “why is 
>> Arcturus GFX9”, etc. Then if things change, we just up the value concretely, 
>> instead of maybe doing a v11 if GFX11 changes things, and only GFX11 ASICs 
>> use those functions/variables.
>>
>>
>>
>> Obviously not high-priority, but maybe something to consider as you continue 
>> to consolidate and remove duplicate code.
>>
>>
>>
>> Kent
>>
>>
>>
>> From: amd-gfx  On Behalf Of Zhao, Yong
>> Sent: Thursday, November 7, 2019 11:57 AM
>> To: Kuehling, Felix ; amd-gfx@lists.freedesktop.org
>> Subject: Re: [PATCH 2/3] drm/amdkfd: only keep release_mem function for 
>> Hawaii
>>
>>
>>
>> Hi Felix,
>>
>>
>>
>> That's because v8 and v7 share the same packet_manager_funcs. In this case, 
>> it is better to keep it as it is.
&g

Re: [PATCH] drm/amdkfd: Use kernel queue v9 functions for v10 (ver2)

[Kuehling, Felix]

Are you sure that setting the SQ_SHADER_TBA_HI__TRAP_EN bit on GFXv9 is 
completely harmless? If the field is not defined, maybe setting the bit 
makes the address invalid. It's probably worth running that through a 
PSDB, which would cover Vega10, Vega20 and Arcturus.

If it actually works, the patch is

Reviewed-by: Felix Kuehling 

Regards,
   Felix

On 2019-11-07 15:34, Zhao, Yong wrote:
> The kernel queue functions for v9 and v10 are the same except
> pm_map_process_v* which have small difference, so they should be reused.
> This eliminates the need of reapplying several patches which were
> applied on v9 but not on v10, such as bigger GWS and more than 2
> SDMA engine support which were introduced on Arcturus.
>
> Change-Id: I2d385961e3c884db14e30b5afc98d0d9e4cb1802
> Signed-off-by: Yong Zhao 
> ---
>   drivers/gpu/drm/amd/amdkfd/Makefile   |   1 -
>   drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.c |   4 +-
>   drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.h |   1 -
>   .../gpu/drm/amd/amdkfd/kfd_kernel_queue_v10.c | 317 --
>   .../gpu/drm/amd/amdkfd/kfd_kernel_queue_v9.c  |  16 +-
>   .../gpu/drm/amd/amdkfd/kfd_packet_manager.c   |   4 +-
>   drivers/gpu/drm/amd/amdkfd/kfd_priv.h |   4 -
>   7 files changed, 14 insertions(+), 333 deletions(-)
>   delete mode 100644 drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v10.c
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/Makefile 
> b/drivers/gpu/drm/amd/amdkfd/Makefile
> index 48155060a57c..017a8b7156da 100644
> --- a/drivers/gpu/drm/amd/amdkfd/Makefile
> +++ b/drivers/gpu/drm/amd/amdkfd/Makefile
> @@ -41,7 +41,6 @@ AMDKFD_FILES:= $(AMDKFD_PATH)/kfd_module.o \
>   $(AMDKFD_PATH)/kfd_kernel_queue_cik.o \
>   $(AMDKFD_PATH)/kfd_kernel_queue_vi.o \
>   $(AMDKFD_PATH)/kfd_kernel_queue_v9.o \
> - $(AMDKFD_PATH)/kfd_kernel_queue_v10.o \
>   $(AMDKFD_PATH)/kfd_packet_manager.o \
>   $(AMDKFD_PATH)/kfd_process_queue_manager.o \
>   $(AMDKFD_PATH)/kfd_device_queue_manager.o \
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.c
> index 11d244891393..0d966408ea87 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.c
> @@ -332,12 +332,10 @@ struct kernel_queue *kernel_queue_init(struct kfd_dev 
> *dev,
>   case CHIP_RAVEN:
>   case CHIP_RENOIR:
>   case CHIP_ARCTURUS:
> - kernel_queue_init_v9(&kq->ops_asic_specific);
> - break;
>   case CHIP_NAVI10:
>   case CHIP_NAVI12:
>   case CHIP_NAVI14:
> - kernel_queue_init_v10(&kq->ops_asic_specific);
> + kernel_queue_init_v9(&kq->ops_asic_specific);
>   break;
>   default:
>   WARN(1, "Unexpected ASIC family %u",
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.h 
> b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.h
> index 365fc674fea4..a7116a939029 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.h
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.h
> @@ -102,6 +102,5 @@ struct kernel_queue {
>   void kernel_queue_init_cik(struct kernel_queue_ops *ops);
>   void kernel_queue_init_vi(struct kernel_queue_ops *ops);
>   void kernel_queue_init_v9(struct kernel_queue_ops *ops);
> -void kernel_queue_init_v10(struct kernel_queue_ops *ops);
>   
>   #endif /* KFD_KERNEL_QUEUE_H_ */
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v10.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v10.c
> deleted file mode 100644
> index bfd6221acae9..
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v10.c
> +++ /dev/null
> @@ -1,317 +0,0 @@
> -/*
> - * Copyright 2018 Advanced Micro Devices, Inc.
> - *
> - * Permission is hereby granted, free of charge, to any person obtaining a
> - * copy of this software and associated documentation files (the "Software"),
> - * to deal in the Software without restriction, including without limitation
> - * the rights to use, copy, modify, merge, publish, distribute, sublicense,
> - * and/or sell copies of the Software, and to permit persons to whom the
> - * Software is furnished to do so, subject to the following conditions:
> - *
> - * The above copyright notice and this permission notice shall be included in
> - * all copies or substantial portions of the Software.
> - *
> - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> - * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> - * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
> - * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
> - * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
> - * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
> - * OTHER DEALINGS IN THE SOFTWARE.
> - *
> - */
> -
> -#include "kfd_kernel_queue.h"
> -#include "kf

RE: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu notifier v4

[Kuehling, Felix]

I had a chat with Jerome yesterday. He pointed out that the new blockable 
parameter can be used to infer whether the MMU notifier is being called  in a 
reclaim operation. So if blockable==true, it should even be safe to take the BO 
reservation lock without problems. I think with that we should be able to 
remove the read-write locking completely and go back to locking (or try-locking 
for blockable==false) the reservation locks in the MMU notifier?

Regards,
  Felix

-Original Message-
From: amd-gfx  On Behalf Of Christian 
König
Sent: Saturday, September 15, 2018 3:47 AM
To: Kuehling, Felix ; Yang, Philip 
; amd-gfx@lists.freedesktop.org; Jerome Glisse 

Subject: Re: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu 
notifier v4

Am 14.09.2018 um 22:21 schrieb Felix Kuehling:
> On 2018-09-14 01:52 PM, Christian König wrote:
>> Am 14.09.2018 um 19:47 schrieb Philip Yang:
>>> On 2018-09-14 03:51 AM, Christian König wrote:
>>>> Am 13.09.2018 um 23:51 schrieb Felix Kuehling:
>>>>> On 2018-09-13 04:52 PM, Philip Yang wrote:
>>>>> [SNIP]
>>>>>>    +    amdgpu_mn_read_unlock(amn);
>>>>>> +
>>>>> amdgpu_mn_read_lock/unlock support recursive locking for multiple 
>>>>> overlapping or nested invalidation ranges. But if you'r locking 
>>>>> and unlocking in the same function. Is that still a concern?
>>> I don't understand the possible recursive case, but
>>> amdgpu_mn_read_lock() still support recursive locking.
>>>> Well the real problem is that unlocking them here won't work.
>>>>
>>>> We need to hold the lock until we are sure that the operation which 
>>>> updates the page tables is completed.
>>>>
>>> The reason for this change is because hmm mirror has 
>>> invalidate_start callback, no invalidate_end callback
>>>
>>> Check mmu_notifier.c and hmm.c again, below is entire logic to 
>>> update CPU page tables and callback:
>>>
>>> mn lock amn->lock is used to protect interval tree access because 
>>> user may submit/register new userptr anytime.
>>> This is same for old and new way.
>>>
>>> step 2 guarantee the GPU operation is done before updating CPU page 
>>> table.
>>>
>>> So I think the change is safe. We don't need hold mn lock until the 
>>> CPU page tables update is completed.
>> No, that isn't even remotely correct. The lock doesn't protects the 
>> interval tree.
>>
>>> Old:
>>>     1. down_read_non_owner(&amn->lock)
>>>     2. loop to handle BOs from node->bos through interval tree
>>> amn->object nodes
>>>     gfx: wait for pending BOs fence operation done, mark user 
>>> pages dirty
>>>     kfd: evict user queues of the process, wait for queue 
>>> unmap/map operation done
>>>     3. update CPU page tables
>>>     4. up_read(&amn->lock)
>>>
>>> New, switch step 3 and 4
>>>     1. down_read_non_owner(&amn->lock)
>>>     2. loop to handle BOs from node->bos through interval tree
>>> amn->object nodes
>>>     gfx: wait for pending BOs fence operation done, mark user 
>>> pages dirty
>>>     kfd: evict user queues of the process, wait for queue 
>>> unmap/map operation done
>>>     3. up_read(&amn->lock)
>>>     4. update CPU page tables
>> The lock is there to make sure that we serialize page table updates 
>> with command submission.
> As I understand it, the idea is to prevent command submission (adding 
> new fences to BOs) while a page table invalidation is in progress.

Yes, exactly.

> But do we really need another lock for this? Wouldn't the 
> re-validation of userptr BOs (currently calling get_user_pages) force 
> synchronization with the ongoing page table invalidation through the 
> mmap_sem or other MM locks?

No and yes. We don't hold any other locks while doing command submission, but I 
expect that HMM has its own mechanism to prevent that.

Since we don't modify amdgpu_mn_lock()/amdgpu_mn_unlock() we are certainly not 
using this mechanism correctly.

Regards,
Christian.
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu notifier v4

[Kuehling, Felix]

So back to my previous question:



>> But do we really need another lock for this? Wouldn't the

>> re-validation of userptr BOs (currently calling get_user_pages) force

>> synchronization with the ongoing page table invalidation through the

>> mmap_sem or other MM locks?

>

> No and yes. We don't hold any other locks while doing command submission, but 
> I expect that HMM has its own mechanism to prevent that.

>

> Since we don't modify amdgpu_mn_lock()/amdgpu_mn_unlock() we are certainly 
> not using this mechanism correctly.

The existing amdgpu_mn_lock/unlock should block the MMU notifier while a 
command submission is in progress. It should also block command submission 
while an MMU notifier is in progress.

What we lose with HMM is the ability to hold a read-lock for the entire 
duration of the invalidate_range_start until invalidate_range_end. As I 
understand it, that lock is meant to prevent new command submissions while the 
page tables are being updated by the kernel. But my point is, that 
get_user_pages or hmm_vma_fault should do the same kind of thing. Before the 
command submission can go ahead, it needs to update the userptr addresses. If 
the page tables are still being updated, it will block there even without 
holding the amdgpu_mn_read_lock.

Regards,
  Felix

From: Koenig, Christian
Sent: Thursday, September 27, 2018 3:00 AM
To: Kuehling, Felix 
Cc: Yang, Philip ; amd-gfx@lists.freedesktop.org; Jerome 
Glisse 
Subject: RE: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu 
notifier v4

No, that won't work. We would still run into lock inversion problems.

What we could do with the scheduler is to turn submissions into dummies if we 
find that the page tables are now outdated.

But that would be really hacky and I'm not sure if that would really work in 
all cases.

Christian.

Am 27.09.2018 08:53 schrieb "Kuehling, Felix" 
mailto:felix.kuehl...@amd.com>>:
I had a chat with Jerome yesterday. He pointed out that the new blockable 
parameter can be used to infer whether the MMU notifier is being called  in a 
reclaim operation. So if blockable==true, it should even be safe to take the BO 
reservation lock without problems. I think with that we should be able to 
remove the read-write locking completely and go back to locking (or try-locking 
for blockable==false) the reservation locks in the MMU notifier?

Regards,
  Felix

-Original Message-
From: amd-gfx 
mailto:amd-gfx-boun...@lists.freedesktop.org>>
 On Behalf Of Christian König
Sent: Saturday, September 15, 2018 3:47 AM
To: Kuehling, Felix mailto:felix.kuehl...@amd.com>>; 
Yang, Philip mailto:philip.y...@amd.com>>; 
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Jerome 
Glisse mailto:j.gli...@gmail.com>>
Subject: Re: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu 
notifier v4

Am 14.09.2018 um 22:21 schrieb Felix Kuehling:
> On 2018-09-14 01:52 PM, Christian König wrote:
>> Am 14.09.2018 um 19:47 schrieb Philip Yang:
>>> On 2018-09-14 03:51 AM, Christian König wrote:
>>>> Am 13.09.2018 um 23:51 schrieb Felix Kuehling:
>>>>> On 2018-09-13 04:52 PM, Philip Yang wrote:
>>>>> [SNIP]
>>>>>>+amdgpu_mn_read_unlock(amn);
>>>>>> +
>>>>> amdgpu_mn_read_lock/unlock support recursive locking for multiple
>>>>> overlapping or nested invalidation ranges. But if you'r locking
>>>>> and unlocking in the same function. Is that still a concern?
>>> I don't understand the possible recursive case, but
>>> amdgpu_mn_read_lock() still support recursive locking.
>>>> Well the real problem is that unlocking them here won't work.
>>>>
>>>> We need to hold the lock until we are sure that the operation which
>>>> updates the page tables is completed.
>>>>
>>> The reason for this change is because hmm mirror has
>>> invalidate_start callback, no invalidate_end callback
>>>
>>> Check mmu_notifier.c and hmm.c again, below is entire logic to
>>> update CPU page tables and callback:
>>>
>>> mn lock amn->lock is used to protect interval tree access because
>>> user may submit/register new userptr anytime.
>>> This is same for old and new way.
>>>
>>> step 2 guarantee the GPU operation is done before updating CPU page
>>> table.
>>>
>>> So I think the change is safe. We don't need hold mn lock until the
>>> CPU page tables update is completed.
>> No, that isn't even remotely correct. The lock doesn't protects the
>> interval tree.
>>
>>> Old:
>>> 1. down_read_non_owner(&

RE: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu notifier v4

[Kuehling, Felix]

> We double check that there wasn't any page table modification while we 
> prepared the submission and restart the whole process when there actually was 
> some update.
>
> The reason why we need to do this is here:
>
>ttm_eu_fence_buffer_objects(&p->ticket, &p->validated, p->fence);
>amdgpu_mn_unlock(p->mn);
>
> Only after the new fence is added to the buffer object we can release the 
> lock so that any invalidation will now block on our command submission to 
> finish before it modifies the page table.

I don’t see why this requires holding the read-lock until invalidate_range_end. 
amdgpu_ttm_tt_affect_userptr gets called while the mn read-lock is held in 
invalidate_range_start notifier.

Regards,
  Felix

From: Koenig, Christian
Sent: Thursday, September 27, 2018 5:27 AM
To: Kuehling, Felix 
Cc: Yang, Philip ; amd-gfx@lists.freedesktop.org; Jerome 
Glisse 
Subject: Re: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu 
notifier v4

That is correct, but take a look what we do when after calling the 
amdgpu_mn_read_lock():


/* No memory allocation is allowed while holding the mn lock */
amdgpu_mn_lock(p->mn);
amdgpu_bo_list_for_each_userptr_entry(e, p->bo_list) {
struct amdgpu_bo *bo = ttm_to_amdgpu_bo(e->tv.bo);

if (amdgpu_ttm_tt_userptr_needs_pages(bo->tbo.ttm)) {
r = -ERESTARTSYS;
goto error_abort;
}
}

We double check that there wasn't any page table modification while we prepared 
the submission and restart the whole process when there actually was some 
update.

The reason why we need to do this is here:

ttm_eu_fence_buffer_objects(&p->ticket, &p->validated, p->fence);
amdgpu_mn_unlock(p->mn);

Only after the new fence is added to the buffer object we can release the lock 
so that any invalidation will now block on our command submission to finish 
before it modifies the page table.

The only other option would be to add the fence first and then check if there 
was any update to the page tables.

The issue with that approach is that adding a fence can't be made undone, so if 
we find that there actually was an update to the page tables we would need to 
somehow turn the CS into a dummy (e.g. overwrite all IBs with NOPs or something 
like that) and still submit it.

Not sure if that is actually possible.

Regards,
Christian.

Am 27.09.2018 um 10:47 schrieb Kuehling, Felix:
So back to my previous question:



>> But do we really need another lock for this? Wouldn't the

>> re-validation of userptr BOs (currently calling get_user_pages) force

>> synchronization with the ongoing page table invalidation through the

>> mmap_sem or other MM locks?

>

> No and yes. We don't hold any other locks while doing command submission, but 
> I expect that HMM has its own mechanism to prevent that.

>

> Since we don't modify amdgpu_mn_lock()/amdgpu_mn_unlock() we are certainly 
> not using this mechanism correctly.

The existing amdgpu_mn_lock/unlock should block the MMU notifier while a 
command submission is in progress. It should also block command submission 
while an MMU notifier is in progress.

What we lose with HMM is the ability to hold a read-lock for the entire 
duration of the invalidate_range_start until invalidate_range_end. As I 
understand it, that lock is meant to prevent new command submissions while the 
page tables are being updated by the kernel. But my point is, that 
get_user_pages or hmm_vma_fault should do the same kind of thing. Before the 
command submission can go ahead, it needs to update the userptr addresses. If 
the page tables are still being updated, it will block there even without 
holding the amdgpu_mn_read_lock.

Regards,
  Felix

From: Koenig, Christian
Sent: Thursday, September 27, 2018 3:00 AM
To: Kuehling, Felix <mailto:felix.kuehl...@amd.com>
Cc: Yang, Philip <mailto:philip.y...@amd.com>; 
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Jerome 
Glisse <mailto:j.gli...@gmail.com>
Subject: RE: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu 
notifier v4

No, that won't work. We would still run into lock inversion problems.

What we could do with the scheduler is to turn submissions into dummies if we 
find that the page tables are now outdated.

But that would be really hacky and I'm not sure if that would really work in 
all cases.

Christian.

Am 27.09.2018 08:53 schrieb "Kuehling, Felix" 
mailto:felix.kuehl...@amd.com>>:
I had a chat with Jerome yesterday. He pointed out that the new blockable 
parameter can be used to infer whether the MMU notifier is being called  in a 
reclaim operation. So if blockable==true, it should even be safe to take the BO 

RE: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu notifier v4

[Kuehling, Felix]

> The problem is here:
>
> ttm_eu_fence_buffer_objects(&p->ticket, &p->validated, p->fence);
> amdgpu_mn_unlock(p->mn);
>
> We need to hold the lock until the fence is added to the reservation object.
>
> Otherwise somebody could have changed the page tables just in the moment 
> between the check of amdgpu_ttm_tt_userptr_needs_pages() and adding the fence 
> to the reservation object.

I’m not planning to change that. I don’t think there is any need to change it.

Regards,
  Felix

From: Koenig, Christian
Sent: Thursday, September 27, 2018 7:24 AM
To: Kuehling, Felix 
Cc: Yang, Philip ; amd-gfx@lists.freedesktop.org; Jerome 
Glisse 
Subject: Re: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu 
notifier v4

Am 27.09.2018 um 13:08 schrieb Kuehling, Felix:
> We double check that there wasn't any page table modification while we 
> prepared the submission and restart the whole process when there actually was 
> some update.
>
> The reason why we need to do this is here:
>
>ttm_eu_fence_buffer_objects(&p->ticket, &p->validated, p->fence);
>amdgpu_mn_unlock(p->mn);
>
> Only after the new fence is added to the buffer object we can release the 
> lock so that any invalidation will now block on our command submission to 
> finish before it modifies the page table.


I don’t see why this requires holding the read-lock until invalidate_range_end. 
amdgpu_ttm_tt_affect_userptr gets called while the mn read-lock is held in 
invalidate_range_start notifier.

That's not related to amdgpu_ttm_tt_affect_userptr(), this function could 
actually be called outside the lock.

The problem is here:

ttm_eu_fence_buffer_objects(&p->ticket, &p->validated, p->fence);
amdgpu_mn_unlock(p->mn);

We need to hold the lock until the fence is added to the reservation object.

Otherwise somebody could have changed the page tables just in the moment 
between the check of amdgpu_ttm_tt_userptr_needs_pages() and adding the fence 
to the reservation object.

Regards,
Christian.




Regards,
  Felix

From: Koenig, Christian
Sent: Thursday, September 27, 2018 5:27 AM
To: Kuehling, Felix <mailto:felix.kuehl...@amd.com>
Cc: Yang, Philip <mailto:philip.y...@amd.com>; 
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Jerome 
Glisse <mailto:j.gli...@gmail.com>
Subject: Re: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu 
notifier v4

That is correct, but take a look what we do when after calling the 
amdgpu_mn_read_lock():



/* No memory allocation is allowed while holding the mn lock */
amdgpu_mn_lock(p->mn);
amdgpu_bo_list_for_each_userptr_entry(e, p->bo_list) {
struct amdgpu_bo *bo = ttm_to_amdgpu_bo(e->tv.bo);

if (amdgpu_ttm_tt_userptr_needs_pages(bo->tbo.ttm)) {
r = -ERESTARTSYS;
goto error_abort;
}
}

We double check that there wasn't any page table modification while we prepared 
the submission and restart the whole process when there actually was some 
update.

The reason why we need to do this is here:


ttm_eu_fence_buffer_objects(&p->ticket, &p->validated, p->fence);
amdgpu_mn_unlock(p->mn);

Only after the new fence is added to the buffer object we can release the lock 
so that any invalidation will now block on our command submission to finish 
before it modifies the page table.

The only other option would be to add the fence first and then check if there 
was any update to the page tables.

The issue with that approach is that adding a fence can't be made undone, so if 
we find that there actually was an update to the page tables we would need to 
somehow turn the CS into a dummy (e.g. overwrite all IBs with NOPs or something 
like that) and still submit it.

Not sure if that is actually possible.

Regards,
Christian.

Am 27.09.2018 um 10:47 schrieb Kuehling, Felix:
So back to my previous question:



>> But do we really need another lock for this? Wouldn't the

>> re-validation of userptr BOs (currently calling get_user_pages) force

>> synchronization with the ongoing page table invalidation through the

>> mmap_sem or other MM locks?

>

> No and yes. We don't hold any other locks while doing command submission, but 
> I expect that HMM has its own mechanism to prevent that.

>

> Since we don't modify amdgpu_mn_lock()/amdgpu_mn_unlock() we are certainly 
> not using this mechanism correctly.

The existing amdgpu_mn_lock/unlock should block the MMU notifier while a 
command submission is in progress. It should also block command submission 
while an MMU notifier is in progress.

What we lose with HMM is the ability to hold a read-lock for the entire 
duration of the invalidate

RE: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu notifier v4

[Kuehling, Felix]

> I’m not planning to change that. I don’t think there is any need to change it.
>
> Yeah, but when HMM doesn't provide both the start and the end hock of the 
> invalidation this way won't work any more.
>
> So we need to find a solution for this,
> Christian.

My whole argument is that you don’t need to hold the read lock until the 
invalidate_range_end. Just read_lock and read_unlock in the 
invalidate_range_start function.

Regards,
  Felix


From: Koenig, Christian
Sent: Thursday, September 27, 2018 9:22 AM
To: Kuehling, Felix 
Cc: Yang, Philip ; amd-gfx@lists.freedesktop.org; Jerome 
Glisse 
Subject: Re: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu 
notifier v4

Am 27.09.2018 um 15:18 schrieb Kuehling, Felix:
> The problem is here:
>
> ttm_eu_fence_buffer_objects(&p->ticket, &p->validated, p->fence);
> amdgpu_mn_unlock(p->mn);
>
> We need to hold the lock until the fence is added to the reservation object.
>
> Otherwise somebody could have changed the page tables just in the moment 
> between the check of amdgpu_ttm_tt_userptr_needs_pages() and adding the fence 
> to the reservation object.


I’m not planning to change that. I don’t think there is any need to change it.

Yeah, but when HMM doesn't provide both the start and the end hock of the 
invalidation this way won't work any more.

So we need to find a solution for this,
Christian.



Regards,
  Felix

From: Koenig, Christian
Sent: Thursday, September 27, 2018 7:24 AM
To: Kuehling, Felix <mailto:felix.kuehl...@amd.com>
Cc: Yang, Philip <mailto:philip.y...@amd.com>; 
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Jerome 
Glisse <mailto:j.gli...@gmail.com>
Subject: Re: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu 
notifier v4

Am 27.09.2018 um 13:08 schrieb Kuehling, Felix:
> We double check that there wasn't any page table modification while we 
> prepared the submission and restart the whole process when there actually was 
> some update.
>
> The reason why we need to do this is here:
>
>ttm_eu_fence_buffer_objects(&p->ticket, &p->validated, p->fence);
>amdgpu_mn_unlock(p->mn);
>
> Only after the new fence is added to the buffer object we can release the 
> lock so that any invalidation will now block on our command submission to 
> finish before it modifies the page table.



I don’t see why this requires holding the read-lock until invalidate_range_end. 
amdgpu_ttm_tt_affect_userptr gets called while the mn read-lock is held in 
invalidate_range_start notifier.

That's not related to amdgpu_ttm_tt_affect_userptr(), this function could 
actually be called outside the lock.

The problem is here:


ttm_eu_fence_buffer_objects(&p->ticket, &p->validated, p->fence);
amdgpu_mn_unlock(p->mn);

We need to hold the lock until the fence is added to the reservation object.

Otherwise somebody could have changed the page tables just in the moment 
between the check of amdgpu_ttm_tt_userptr_needs_pages() and adding the fence 
to the reservation object.

Regards,
Christian.





Regards,
  Felix

From: Koenig, Christian
Sent: Thursday, September 27, 2018 5:27 AM
To: Kuehling, Felix <mailto:felix.kuehl...@amd.com>
Cc: Yang, Philip <mailto:philip.y...@amd.com>; 
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Jerome 
Glisse <mailto:j.gli...@gmail.com>
Subject: Re: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu 
notifier v4

That is correct, but take a look what we do when after calling the 
amdgpu_mn_read_lock():




/* No memory allocation is allowed while holding the mn lock */
amdgpu_mn_lock(p->mn);
amdgpu_bo_list_for_each_userptr_entry(e, p->bo_list) {
struct amdgpu_bo *bo = ttm_to_amdgpu_bo(e->tv.bo);

if (amdgpu_ttm_tt_userptr_needs_pages(bo->tbo.ttm)) {
r = -ERESTARTSYS;
goto error_abort;
}
}

We double check that there wasn't any page table modification while we prepared 
the submission and restart the whole process when there actually was some 
update.

The reason why we need to do this is here:



ttm_eu_fence_buffer_objects(&p->ticket, &p->validated, p->fence);
amdgpu_mn_unlock(p->mn);

Only after the new fence is added to the buffer object we can release the lock 
so that any invalidation will now block on our command submission to finish 
before it modifies the page table.

The only other option would be to add the fence first and then check if there 
was any update to the page tables.

The issue with that approach is that adding a fence can't be made undone, so if 
we find that there actually was an update to the page tables we would need to 
s

RE: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu notifier v4

[Kuehling, Felix]

> In this case you can end up accessing pages which are invalidated while 
> get_user_pages is in process.

What's the sequence of events you have in mind? Something like this?


  *   Page table is updated and triggers MMU notifier
  *   amdgpu MMU notifier runs and waits for pending CS to finish while holding 
the read lock
  *   New CS starts just after invalidate_range_start MMU notifier finishes but 
before the page table update is done
  *   get_user_pages returns outdated physical addresses

I hope that's not actually possible and that get_user_pages or hmm_vma_fault 
would block until the page table update is done. That is, 
invalidate_range_start marks the start of a page table update, and while that 
update is in progress, get_user_pages or hmm_vma_fault block. Jerome, can you 
comment on that?

Thanks,
  Felix

From: Koenig, Christian
Sent: Thursday, September 27, 2018 9:59 AM
To: Kuehling, Felix 
Cc: Yang, Philip ; amd-gfx@lists.freedesktop.org; Jerome 
Glisse 
Subject: RE: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu 
notifier v4

Yeah I understand that, but again that won't work.

In this case you can end up accessing pages which are invalidated while 
get_user_pages is in process.

Christian.

Am 27.09.2018 15:41 schrieb "Kuehling, Felix" 
mailto:felix.kuehl...@amd.com>>:
> I'm not planning to change that. I don't think there is any need to change it.
>
> Yeah, but when HMM doesn't provide both the start and the end hock of the 
> invalidation this way won't work any more.
>
> So we need to find a solution for this,
> Christian.

My whole argument is that you don't need to hold the read lock until the 
invalidate_range_end. Just read_lock and read_unlock in the 
invalidate_range_start function.

Regards,
  Felix

From: Koenig, Christian
Sent: Thursday, September 27, 2018 9:22 AM
To: Kuehling, Felix mailto:felix.kuehl...@amd.com>>
Cc: Yang, Philip mailto:philip.y...@amd.com>>; 
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Jerome 
Glisse mailto:j.gli...@gmail.com>>
Subject: Re: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu 
notifier v4

Am 27.09.2018 um 15:18 schrieb Kuehling, Felix:
> The problem is here:
>
> ttm_eu_fence_buffer_objects(&p->ticket, &p->validated, p->fence);
> amdgpu_mn_unlock(p->mn);
>
> We need to hold the lock until the fence is added to the reservation object.
>
> Otherwise somebody could have changed the page tables just in the moment 
> between the check of amdgpu_ttm_tt_userptr_needs_pages() and adding the fence 
> to the reservation object.

I'm not planning to change that. I don't think there is any need to change it.

Yeah, but when HMM doesn't provide both the start and the end hock of the 
invalidation this way won't work any more.

So we need to find a solution for this,
Christian.


Regards,
  Felix

From: Koenig, Christian
Sent: Thursday, September 27, 2018 7:24 AM
To: Kuehling, Felix <mailto:felix.kuehl...@amd.com>
Cc: Yang, Philip <mailto:philip.y...@amd.com>; 
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Jerome 
Glisse <mailto:j.gli...@gmail.com>
Subject: Re: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu 
notifier v4

Am 27.09.2018 um 13:08 schrieb Kuehling, Felix:
> We double check that there wasn't any page table modification while we 
> prepared the submission and restart the whole process when there actually was 
> some update.
>
> The reason why we need to do this is here:
>
>ttm_eu_fence_buffer_objects(&p->ticket, &p->validated, p->fence);
>amdgpu_mn_unlock(p->mn);
>
> Only after the new fence is added to the buffer object we can release the 
> lock so that any invalidation will now block on our command submission to 
> finish before it modifies the page table.


I don't see why this requires holding the read-lock until invalidate_range_end. 
amdgpu_ttm_tt_affect_userptr gets called while the mn read-lock is held in 
invalidate_range_start notifier.

That's not related to amdgpu_ttm_tt_affect_userptr(), this function could 
actually be called outside the lock.

The problem is here:

ttm_eu_fence_buffer_objects(&p->ticket, &p->validated, p->fence);
amdgpu_mn_unlock(p->mn);

We need to hold the lock until the fence is added to the reservation object.

Otherwise somebody could have changed the page tables just in the moment 
between the check of amdgpu_ttm_tt_userptr_needs_pages() and adding the fence 
to the reservation object.

Regards,
Christian.




Regards,
  Felix

From: Koenig, Christian
Sent: Thursday, September 27, 2018 5:27 AM
To: Kuehling, Felix <mailto:felix.kuehl...@amd.com>
Cc: Yang, Philip <mailto:philip.y...@amd.com>; 
amd-gfx@lists.freed

RE: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu notifier v4

[Kuehling, Felix]

I think the answer is here: 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/vm/hmm.rst#n216

Regards,
  Felix

From: Koenig, Christian
Sent: Thursday, September 27, 2018 10:30 AM
To: Kuehling, Felix 
Cc: j.gli...@gmail.com; Yang, Philip ; 
amd-gfx@lists.freedesktop.org
Subject: RE: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu 
notifier v4

At least with get_user_pages() that is perfectly possible.

For HMM it could be that this is prevented somehow.

Christian.

Am 27.09.2018 16:27 schrieb "Kuehling, Felix" 
mailto:felix.kuehl...@amd.com>>:
> In this case you can end up accessing pages which are invalidated while 
> get_user_pages is in process.

What's the sequence of events you have in mind? Something like this?


  *   Page table is updated and triggers MMU notifier
  *   amdgpu MMU notifier runs and waits for pending CS to finish while holding 
the read lock
  *   New CS starts just after invalidate_range_start MMU notifier finishes but 
before the page table update is done
  *   get_user_pages returns outdated physical addresses

I hope that's not actually possible and that get_user_pages or hmm_vma_fault 
would block until the page table update is done. That is, 
invalidate_range_start marks the start of a page table update, and while that 
update is in progress, get_user_pages or hmm_vma_fault block. Jerome, can you 
comment on that?

Thanks,
  Felix

From: Koenig, Christian
Sent: Thursday, September 27, 2018 9:59 AM
To: Kuehling, Felix mailto:felix.kuehl...@amd.com>>
Cc: Yang, Philip mailto:philip.y...@amd.com>>; 
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Jerome 
Glisse mailto:j.gli...@gmail.com>>
Subject: RE: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu 
notifier v4

Yeah I understand that, but again that won't work.

In this case you can end up accessing pages which are invalidated while 
get_user_pages is in process.

Christian.

Am 27.09.2018 15:41 schrieb "Kuehling, Felix" 
mailto:felix.kuehl...@amd.com>>:
> I'm not planning to change that. I don't think there is any need to change it.
>
> Yeah, but when HMM doesn't provide both the start and the end hock of the 
> invalidation this way won't work any more.
>
> So we need to find a solution for this,
> Christian.

My whole argument is that you don't need to hold the read lock until the 
invalidate_range_end. Just read_lock and read_unlock in the 
invalidate_range_start function.

Regards,
  Felix

From: Koenig, Christian
Sent: Thursday, September 27, 2018 9:22 AM
To: Kuehling, Felix mailto:felix.kuehl...@amd.com>>
Cc: Yang, Philip mailto:philip.y...@amd.com>>; 
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Jerome 
Glisse mailto:j.gli...@gmail.com>>
Subject: Re: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu 
notifier v4

Am 27.09.2018 um 15:18 schrieb Kuehling, Felix:
> The problem is here:
>
> ttm_eu_fence_buffer_objects(&p->ticket, &p->validated, p->fence);
> amdgpu_mn_unlock(p->mn);
>
> We need to hold the lock until the fence is added to the reservation object.
>
> Otherwise somebody could have changed the page tables just in the moment 
> between the check of amdgpu_ttm_tt_userptr_needs_pages() and adding the fence 
> to the reservation object.
I'm not planning to change that. I don't think there is any need to change it.

Yeah, but when HMM doesn't provide both the start and the end hock of the 
invalidation this way won't work any more.

So we need to find a solution for this,
Christian.

Regards,
  Felix

From: Koenig, Christian
Sent: Thursday, September 27, 2018 7:24 AM
To: Kuehling, Felix <mailto:felix.kuehl...@amd.com>
Cc: Yang, Philip <mailto:philip.y...@amd.com>; 
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Jerome 
Glisse <mailto:j.gli...@gmail.com>
Subject: Re: [PATCH] drm/amdgpu: use HMM mirror callback to replace mmu 
notifier v4

Am 27.09.2018 um 13:08 schrieb Kuehling, Felix:
> We double check that there wasn't any page table modification while we 
> prepared the submission and restart the whole process when there actually was 
> some update.
>
> The reason why we need to do this is here:
>
>ttm_eu_fence_buffer_objects(&p->ticket, &p->validated, p->fence);
>amdgpu_mn_unlock(p->mn);
>
> Only after the new fence is added to the buffer object we can release the 
> lock so that any invalidation will now block on our command submission to 
> finish before it modifies the page table.

I don't see why this requires holding the read-lock until invalidate_range_end. 
amdgpu_ttm_tt_affect_userptr gets called while the mn read-lock is held in 
invalidate_range_star

Re: [PATCH] drm/amdkfd: Fix incorrect use of process->mm

[Kuehling, Felix]

Hi Alex,

If it's not too late, I'd like to get this into 4.19. Sorry I missed this fix 
earlier.

Regards,
  Felix

____
From: Kuehling, Felix
Sent: Tuesday, October 2, 2018 6:41:12 PM
To: amd-gfx@lists.freedesktop.org
Cc: oded.gab...@gmail.com; Kuehling, Felix
Subject: [PATCH] drm/amdkfd: Fix incorrect use of process->mm

This mm_struct pointer should never be dereferenced. If running in
a user thread, just use current->mm. If running in a kernel worker
use get_task_mm to get a safe reference to the mm_struct.

Signed-off-by: Felix Kuehling 
---
 .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.c  | 37 +-
 1 file changed, 29 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c 
b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
index d6af31c..3bc0651d 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
@@ -358,8 +358,8 @@ static int create_compute_queue_nocpsch(struct 
device_queue_manager *dqm,
struct queue *q,
struct qcm_process_device *qpd)
 {
-   int retval;
struct mqd_manager *mqd_mgr;
+   int retval;

mqd_mgr = dqm->ops.get_mqd_manager(dqm, KFD_MQD_TYPE_COMPUTE);
if (!mqd_mgr)
@@ -387,8 +387,12 @@ static int create_compute_queue_nocpsch(struct 
device_queue_manager *dqm,
if (!q->properties.is_active)
return 0;

-   retval = mqd_mgr->load_mqd(mqd_mgr, q->mqd, q->pipe, q->queue,
-   &q->properties, q->process->mm);
+   if (WARN(q->process->mm != current->mm,
+"should only run in user thread"))
+   retval = -EFAULT;
+   else
+   retval = mqd_mgr->load_mqd(mqd_mgr, q->mqd, q->pipe, q->queue,
+  &q->properties, current->mm);
if (retval)
goto out_uninit_mqd;

@@ -545,9 +549,15 @@ static int update_queue(struct device_queue_manager *dqm, 
struct queue *q)
retval = map_queues_cpsch(dqm);
else if (q->properties.is_active &&
 (q->properties.type == KFD_QUEUE_TYPE_COMPUTE ||
- q->properties.type == KFD_QUEUE_TYPE_SDMA))
-   retval = mqd_mgr->load_mqd(mqd_mgr, q->mqd, q->pipe, q->queue,
-  &q->properties, q->process->mm);
+ q->properties.type == KFD_QUEUE_TYPE_SDMA)) {
+   if (WARN(q->process->mm != current->mm,
+"should only run in user thread"))
+   retval = -EFAULT;
+   else
+   retval = mqd_mgr->load_mqd(mqd_mgr, q->mqd,
+  q->pipe, q->queue,
+  &q->properties, current->mm);
+   }

 out_unlock:
dqm_unlock(dqm);
@@ -653,6 +663,7 @@ static int evict_process_queues_cpsch(struct 
device_queue_manager *dqm,
 static int restore_process_queues_nocpsch(struct device_queue_manager *dqm,
  struct qcm_process_device *qpd)
 {
+   struct mm_struct *mm = NULL;
struct queue *q;
struct mqd_manager *mqd_mgr;
struct kfd_process_device *pdd;
@@ -686,6 +697,15 @@ static int restore_process_queues_nocpsch(struct 
device_queue_manager *dqm,
kfd_flush_tlb(pdd);
}

+   /* Take a safe reference to the mm_struct, which may otherwise
+* disappear even while the kfd_process is still referenced.
+*/
+   mm = get_task_mm(pdd->process->lead_thread);
+   if (!mm) {
+   retval = -EFAULT;
+   goto out;
+   }
+
/* activate all active queues on the qpd */
list_for_each_entry(q, &qpd->queues_list, list) {
if (!q->properties.is_evicted)
@@ -700,14 +720,15 @@ static int restore_process_queues_nocpsch(struct 
device_queue_manager *dqm,
q->properties.is_evicted = false;
q->properties.is_active = true;
retval = mqd_mgr->load_mqd(mqd_mgr, q->mqd, q->pipe,
-  q->queue, &q->properties,
-  q->process->mm);
+  q->queue, &q->properties, mm);
if (retval)
goto out;
dqm->queue_count++;
}
qpd->evicted = 0;
 out:
+   if (mm)
+   mmput(mm);
dqm_unlock(dqm);
return retval;
 }
--
2.7.4

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 3/3] drm/amdkfd: Add proper prefix to functions

[Kuehling, Felix]

On 2018-10-18 6:03 p.m., Deucher, Alexander wrote:
>
> Series is:
>
> Reviewed-by: Alex Deucher 
>

Reviewed-by: Felix Kuehling 


as well.


> 
> *From:* amd-gfx  on behalf of
> Lin, Amber 
> *Sent:* Thursday, October 18, 2018 5:15:24 PM
> *To:* amd-gfx@lists.freedesktop.org
> *Cc:* Lin, Amber
> *Subject:* [PATCH 3/3] drm/amdkfd: Add proper prefix to functions
>  
> Add amdgpu_amdkfd_ prefix to amdgpu functions served for amdkfd usage.
>
> Signed-off-by: Amber Lin 
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c  | 12 ++--
>  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h  | 12 ++--
>  drivers/gpu/drm/amd/amdkfd/kfd_chardev.c    |  4 ++--
>  drivers/gpu/drm/amd/amdkfd/kfd_crat.c   |  4 ++--
>  drivers/gpu/drm/amd/amdkfd/kfd_device.c |  6 +++---
>  drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager.c    |  2 +-
>  drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c |  4 ++--
>  drivers/gpu/drm/amd/amdkfd/kfd_topology.c   |  8 
>  8 files changed, 26 insertions(+), 26 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
> index c31a884..4e384ab 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
> @@ -268,7 +268,7 @@ void amdgpu_amdkfd_gpu_reset(struct kgd_dev *kgd)
>  amdgpu_device_gpu_recover(adev, NULL);
>  }
>  
> -int alloc_gtt_mem(struct kgd_dev *kgd, size_t size,
> +int amdgpu_amdkfd_alloc_gtt_mem(struct kgd_dev *kgd, size_t size,
>  void **mem_obj, uint64_t *gpu_addr,
>  void **cpu_ptr, bool mqd_gfx9)
>  {
> @@ -340,7 +340,7 @@ int alloc_gtt_mem(struct kgd_dev *kgd, size_t size,
>  return r;
>  }
>  
> -void free_gtt_mem(struct kgd_dev *kgd, void *mem_obj)
> +void amdgpu_amdkfd_free_gtt_mem(struct kgd_dev *kgd, void *mem_obj)
>  {
>  struct amdgpu_bo *bo = (struct amdgpu_bo *) mem_obj;
>  
> @@ -351,7 +351,7 @@ void free_gtt_mem(struct kgd_dev *kgd, void *mem_obj)
>  amdgpu_bo_unref(&(bo));
>  }
>  
> -void get_local_mem_info(struct kgd_dev *kgd,
> +void amdgpu_amdkfd_get_local_mem_info(struct kgd_dev *kgd,
>  struct kfd_local_mem_info *mem_info)
>  {
>  struct amdgpu_device *adev = (struct amdgpu_device *)kgd;
> @@ -383,7 +383,7 @@ void get_local_mem_info(struct kgd_dev *kgd,
>  mem_info->mem_clk_max = 100;
>  }
>  
> -uint64_t get_gpu_clock_counter(struct kgd_dev *kgd)
> +uint64_t amdgpu_amdkfd_get_gpu_clock_counter(struct kgd_dev *kgd)
>  {
>  struct amdgpu_device *adev = (struct amdgpu_device *)kgd;
>  
> @@ -392,7 +392,7 @@ uint64_t get_gpu_clock_counter(struct kgd_dev *kgd)
>  return 0;
>  }
>  
> -uint32_t get_max_engine_clock_in_mhz(struct kgd_dev *kgd)
> +uint32_t amdgpu_amdkfd_get_max_engine_clock_in_mhz(struct kgd_dev *kgd)
>  {
>  struct amdgpu_device *adev = (struct amdgpu_device *)kgd;
>  
> @@ -405,7 +405,7 @@ uint32_t get_max_engine_clock_in_mhz(struct
> kgd_dev *kgd)
>  return 100;
>  }
>  
> -void get_cu_info(struct kgd_dev *kgd, struct kfd_cu_info *cu_info)
> +void amdgpu_amdkfd_get_cu_info(struct kgd_dev *kgd, struct
> kfd_cu_info *cu_info)
>  {
>  struct amdgpu_device *adev = (struct amdgpu_device *)kgd;
>  struct amdgpu_cu_info acu_info = adev->gfx.cu_info;
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
> index 8e0d4f7..69cc9a9 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
> @@ -134,16 +134,16 @@ int amdgpu_amdkfd_post_reset(struct
> amdgpu_device *adev);
>  void amdgpu_amdkfd_gpu_reset(struct kgd_dev *kgd);
>  
>  /* Shared API */
> -int alloc_gtt_mem(struct kgd_dev *kgd, size_t size,
> +int amdgpu_amdkfd_alloc_gtt_mem(struct kgd_dev *kgd, size_t size,
>  void **mem_obj, uint64_t *gpu_addr,
>  void **cpu_ptr, bool mqd_gfx9);
> -void free_gtt_mem(struct kgd_dev *kgd, void *mem_obj);
> -void get_local_mem_info(struct kgd_dev *kgd,
> +void amdgpu_amdkfd_free_gtt_mem(struct kgd_dev *kgd, void *mem_obj);
> +void amdgpu_amdkfd_get_local_mem_info(struct kgd_dev *kgd,
>  struct kfd_local_mem_info *mem_info);
> -uint64_t get_gpu_clock_counter(struct kgd_dev *kgd);
> +uint64_t amdgpu_amdkfd_get_gpu_clock_counter(struct kgd_dev *kgd);
>  
> -uint32_t get_max_engine_clock_in_mhz(struct kgd_dev *kgd);
> -void get_cu_info(struct kgd_dev *kgd, struct kfd_cu_info *cu_info);
> +uint32_t amdgpu_amdkfd_get_max_engine_clock_in_mhz(struct kgd_dev *kgd);
> +void amdgpu_amdkfd_get_cu_info(struct kgd_dev *kgd, struct
> kfd_cu_info *cu_info);
>  uint64_t amdgpu_amdkfd_get_vram_usage(struct kgd_dev *kgd);
>  uint64_t amdgpu_amdkfd_get_hive_id(struct kgd_dev *kgd);
>  
> diff --git a/driver

Re: [PATCH 3/3] drm/amdkfd: Use functions from amdgpu for setting up page table base

[Kuehling, Felix]

On 2018-10-18 5:59 p.m., wrote:
>
> Please include a patch description on 2 and 3, with that fixed, series is:
>
> Reviewed-by: Alex Deucher 
>

Reviewed-by: Felix Kuehling 


> 
> *From:* Zhao, Yong
> *Sent:* Thursday, October 18, 2018 5:13:04 PM
> *To:* amd-gfx@lists.freedesktop.org; brahma_sw_dev
> *Cc:* Zhao, Yong
> *Subject:* [PATCH 3/3] drm/amdkfd: Use functions from amdgpu for
> setting up page table base
>  
> Change-Id: I42eb2e41ce21b4a6ea0c8394dcc762ee92b2ca5e
> Signed-off-by: Yong Zhao 
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c | 12 +++-
>  1 file changed, 3 insertions(+), 9 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c
> index 4b79639..223bbc1 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c
> @@ -46,6 +46,7 @@
>  #include "v9_structs.h"
>  #include "soc15.h"
>  #include "soc15d.h"
> +#include "gmc_v9_0.h"
>  
>  /* HACK: MMHUB and GC both have VM-related register with the same
>   * names but different offsets. Define the MMHUB register we need here
> @@ -59,11 +60,6 @@
>  #define mmMMHUB_VM_INVALIDATE_ENG16_ACK 0x0705
>  #define mmMMHUB_VM_INVALIDATE_ENG16_ACK_BASE_IDX    0
>  
> -#define mmMMHUB_VM_CONTEXT0_PAGE_TABLE_BASE_ADDR_LO32  0x072b
> -#define mmMMHUB_VM_CONTEXT0_PAGE_TABLE_BASE_ADDR_LO32_BASE_IDX 0
> -#define mmMMHUB_VM_CONTEXT0_PAGE_TABLE_BASE_ADDR_HI32  0x072c
> -#define mmMMHUB_VM_CONTEXT0_PAGE_TABLE_BASE_ADDR_HI32_BASE_IDX 0
> -
>  #define mmMMHUB_VM_INVALIDATE_ENG16_ADDR_RANGE_LO32 0x0727
>  #define mmMMHUB_VM_INVALIDATE_ENG16_ADDR_RANGE_LO32_BASE_IDX    0
>  #define mmMMHUB_VM_INVALIDATE_ENG16_ADDR_RANGE_HI32 0x0728
> @@ -1018,9 +1014,7 @@ static void
> set_vm_context_page_table_base(struct kgd_dev *kgd, uint32_t vmid,
>   * now, all processes share the same address space size, like
>   * on GFX8 and older.
>   */
> -   WREG32(SOC15_REG_OFFSET(MMHUB, 0,
> mmMMHUB_VM_CONTEXT0_PAGE_TABLE_BASE_ADDR_LO32) + (vmid*2),
> lower_32_bits(base));
> -   WREG32(SOC15_REG_OFFSET(MMHUB, 0,
> mmMMHUB_VM_CONTEXT0_PAGE_TABLE_BASE_ADDR_HI32) + (vmid*2),
> upper_32_bits(base));
> +   mmhub_v1_0_setup_vm_pt_regs(adev, vmid, base);
>  
> -   WREG32(SOC15_REG_OFFSET(GC, 0,
> mmVM_CONTEXT0_PAGE_TABLE_BASE_ADDR_LO32) + (vmid*2), lower_32_bits(base));
> -   WREG32(SOC15_REG_OFFSET(GC, 0,
> mmVM_CONTEXT0_PAGE_TABLE_BASE_ADDR_HI32) + (vmid*2), upper_32_bits(base));
> +   gfxhub_v1_0_setup_vm_pt_regs(adev, vmid, base);
>  }
> -- 
> 2.7.4
>
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdgpu: fix sdma v4 ring is disabled accidently

[Kuehling, Felix]

[+Christian]

Should the buffer funcs also use the paging ring? I think that would be
important for being able to clear page tables or migrating a BO while
handling a page fault.

Regards,
  Felix

On 2018-10-19 3:13 p.m., Yang, Philip wrote:
> For sdma v4, there is bug caused by
> commit d4e869b6b5d6 ("drm/amdgpu: add ring test for page queue")'
>
> local variable ring is reused and changed, so 
> amdgpu_ttm_set_buffer_funcs_status(adev, true)
> is skipped accidently. As a result, amdgpu_fill_buffer() will fail, kernel 
> message:
>
> [drm:amdgpu_fill_buffer [amdgpu]] *ERROR* Trying to clear memory with ring 
> turned off.
> [   25.260444] [drm:amdgpu_fill_buffer [amdgpu]] *ERROR* Trying to clear 
> memory with ring turned off.
> [   25.260627] [drm:amdgpu_fill_buffer [amdgpu]] *ERROR* Trying to clear 
> memory with ring turned off.
> [   25.290119] [drm:amdgpu_fill_buffer [amdgpu]] *ERROR* Trying to clear 
> memory with ring turned off.
> [   25.290370] [drm:amdgpu_fill_buffer [amdgpu]] *ERROR* Trying to clear 
> memory with ring turned off.
> [   25.319971] [drm:amdgpu_fill_buffer [amdgpu]] *ERROR* Trying to clear 
> memory with ring turned off.
> [   25.320486] amdgpu :19:00.0: [mmhub] VMC page fault (src_id:0 ring:154 
> vmid:8 pasid:32768, for process  pid 0 thread  pid 0)
> [   25.320533] amdgpu :19:00.0:   in page starting at address 
> 0x from 18
> [   25.320563] amdgpu :19:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00800134
>
> Change-Id: Idacdf8e60557edb0a4a499aa4051b75d87ce4091
> Signed-off-by: Philip Yang 
> ---
>  drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c | 7 ---
>  1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c 
> b/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c
> index ede149a..cd368ac 100644
> --- a/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c
> @@ -1151,10 +1151,11 @@ static int sdma_v4_0_start(struct amdgpu_device *adev)
>   }
>  
>   if (adev->sdma.has_page_queue) {
> - ring = &adev->sdma.instance[i].page;
> - r = amdgpu_ring_test_ring(ring);
> + struct amdgpu_ring *page = &adev->sdma.instance[i].page;
> +
> + r = amdgpu_ring_test_ring(page);
>   if (r) {
> - ring->ready = false;
> + page->ready = false;
>   return r;
>   }
>   }
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH v2] drm/amdkfd: Add proper prefix to functions

[Kuehling, Felix]

On 2018-10-19 11:15 a.m., Lin, Amber wrote:
> Add amdgpu_amdkfd_ prefix to amdgpu functions served for amdkfd usage.
>
> v2: fix indentation
>
> Signed-off-by: Amber Lin 

Reviewed-by: Felix Kuehling 


> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c  | 18 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h  | 20 ++--
>  drivers/gpu/drm/amd/amdkfd/kfd_chardev.c|  4 ++--
>  drivers/gpu/drm/amd/amdkfd/kfd_crat.c   |  4 ++--
>  drivers/gpu/drm/amd/amdkfd/kfd_device.c |  6 +++---
>  drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager.c|  2 +-
>  drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c |  4 ++--
>  drivers/gpu/drm/amd/amdkfd/kfd_topology.c   |  8 
>  8 files changed, 33 insertions(+), 33 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
> index c31a884..bce5f77 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
> @@ -268,9 +268,9 @@ void amdgpu_amdkfd_gpu_reset(struct kgd_dev *kgd)
>   amdgpu_device_gpu_recover(adev, NULL);
>  }
>  
> -int alloc_gtt_mem(struct kgd_dev *kgd, size_t size,
> - void **mem_obj, uint64_t *gpu_addr,
> - void **cpu_ptr, bool mqd_gfx9)
> +int amdgpu_amdkfd_alloc_gtt_mem(struct kgd_dev *kgd, size_t size,
> + void **mem_obj, uint64_t *gpu_addr,
> + void **cpu_ptr, bool mqd_gfx9)
>  {
>   struct amdgpu_device *adev = (struct amdgpu_device *)kgd;
>   struct amdgpu_bo *bo = NULL;
> @@ -340,7 +340,7 @@ int alloc_gtt_mem(struct kgd_dev *kgd, size_t size,
>   return r;
>  }
>  
> -void free_gtt_mem(struct kgd_dev *kgd, void *mem_obj)
> +void amdgpu_amdkfd_free_gtt_mem(struct kgd_dev *kgd, void *mem_obj)
>  {
>   struct amdgpu_bo *bo = (struct amdgpu_bo *) mem_obj;
>  
> @@ -351,8 +351,8 @@ void free_gtt_mem(struct kgd_dev *kgd, void *mem_obj)
>   amdgpu_bo_unref(&(bo));
>  }
>  
> -void get_local_mem_info(struct kgd_dev *kgd,
> - struct kfd_local_mem_info *mem_info)
> +void amdgpu_amdkfd_get_local_mem_info(struct kgd_dev *kgd,
> +   struct kfd_local_mem_info *mem_info)
>  {
>   struct amdgpu_device *adev = (struct amdgpu_device *)kgd;
>   uint64_t address_mask = adev->dev->dma_mask ? ~*adev->dev->dma_mask :
> @@ -383,7 +383,7 @@ void get_local_mem_info(struct kgd_dev *kgd,
>   mem_info->mem_clk_max = 100;
>  }
>  
> -uint64_t get_gpu_clock_counter(struct kgd_dev *kgd)
> +uint64_t amdgpu_amdkfd_get_gpu_clock_counter(struct kgd_dev *kgd)
>  {
>   struct amdgpu_device *adev = (struct amdgpu_device *)kgd;
>  
> @@ -392,7 +392,7 @@ uint64_t get_gpu_clock_counter(struct kgd_dev *kgd)
>   return 0;
>  }
>  
> -uint32_t get_max_engine_clock_in_mhz(struct kgd_dev *kgd)
> +uint32_t amdgpu_amdkfd_get_max_engine_clock_in_mhz(struct kgd_dev *kgd)
>  {
>   struct amdgpu_device *adev = (struct amdgpu_device *)kgd;
>  
> @@ -405,7 +405,7 @@ uint32_t get_max_engine_clock_in_mhz(struct kgd_dev *kgd)
>   return 100;
>  }
>  
> -void get_cu_info(struct kgd_dev *kgd, struct kfd_cu_info *cu_info)
> +void amdgpu_amdkfd_get_cu_info(struct kgd_dev *kgd, struct kfd_cu_info 
> *cu_info)
>  {
>   struct amdgpu_device *adev = (struct amdgpu_device *)kgd;
>   struct amdgpu_cu_info acu_info = adev->gfx.cu_info;
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
> index 8e0d4f7..bcf587b 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
> @@ -134,16 +134,16 @@ int amdgpu_amdkfd_post_reset(struct amdgpu_device 
> *adev);
>  void amdgpu_amdkfd_gpu_reset(struct kgd_dev *kgd);
>  
>  /* Shared API */
> -int alloc_gtt_mem(struct kgd_dev *kgd, size_t size,
> - void **mem_obj, uint64_t *gpu_addr,
> - void **cpu_ptr, bool mqd_gfx9);
> -void free_gtt_mem(struct kgd_dev *kgd, void *mem_obj);
> -void get_local_mem_info(struct kgd_dev *kgd,
> - struct kfd_local_mem_info *mem_info);
> -uint64_t get_gpu_clock_counter(struct kgd_dev *kgd);
> -
> -uint32_t get_max_engine_clock_in_mhz(struct kgd_dev *kgd);
> -void get_cu_info(struct kgd_dev *kgd, struct kfd_cu_info *cu_info);
> +int amdgpu_amdkfd_alloc_gtt_mem(struct kgd_dev *kgd, size_t size,
> + void **mem_obj, uint64_t *gpu_addr,
> + void **cpu_ptr, bool mqd_gfx9);
> +void amdgpu_amdkfd_free_gtt_mem(struct kgd_dev *kgd, void *mem_obj);
> +void amdgpu_amdkfd_get_local_mem_info(struct kgd_dev *kgd,
> +   struct kfd_local_mem_info *mem_info);
> +uint64_t amdgpu_amdkfd_get_gpu_clock_counter(struct kgd_dev *kgd);
> +
> +uint32_t amdgpu_amdkfd_get_max_engine_clock_in_mhz(struct kgd_dev *kgd);
> +void amdgpu

Re: [PATCH] drm/amdgpu: fix a missing-check bug

[Kuehling, Felix]

The BIOS signature check does not guarantee integrity of the BIOS image
either way. As I understand it, the signature is just a magic number.
It's not a cryptographic signature. The check is just a sanity check.
Therefore this change doesn't add any meaningful protection against the
scenario you described.

Regards,
  Felix

On 2018-10-20 4:57 p.m., Wenwen Wang wrote:
> In amdgpu_read_bios_from_rom(), the header of the VBIOS is firstly copied
> to 'header' from an IO memory region through
> amdgpu_asic_read_bios_from_rom(). Then the header is checked to see whether
> it is a valid header. If yes, the whole VBIOS, including the header, is
> then copied to 'adev->bios'. The problem here is that no check is enforced
> on the header after the second copy. Given that the device also has the
> permission to access the IO memory region, it is possible for a malicious
> device controlled by an attacker to modify the header between these two
> copies. By doing so, the attacker can supply compromised VBIOS, which can
> cause undefined behavior of the kernel and introduce potential security
> issues.
>
> This patch rewrites the header in 'adev->bios' using the header acquired in
> the first copy.
>
> Signed-off-by: Wenwen Wang 
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c
> index a5df80d..ac701f4 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c
> @@ -181,6 +181,8 @@ static bool amdgpu_read_bios_from_rom(struct 
> amdgpu_device *adev)
>   /* read complete BIOS */
>   amdgpu_asic_read_bios_from_rom(adev, adev->bios, len);
>  
> + memcpy(adev->bios, header, AMD_VBIOS_SIGNATURE_END);
> +
>   if (!check_atom_bios(adev->bios, len)) {
>   kfree(adev->bios);
>   return false;
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 1/3] drm/amdkfd: Remove unnecessary register setting when invalidating tlb in kfd

[Kuehling, Felix]

Patch 1 is Reviewed-by: Felix Kuehling 

Patch 2: I'm not sure we need the "lock" parameter and the invalidation
engine parameter. If we're serious about consolidating TLB invalidation
between amdgpu and KFD, I think we should use the same invalidation
engine and the same lock. Then you also don't need to take the
adev->srbm_mutex any more in write_vmid_invalidate_request, which we
were abusing for this purpose.

Regards,
  Felix


On 2018-10-22 2:41 p.m., Zhao, Yong wrote:
> Those register settings have been done in gfxhub_v1_0_program_invalidation()
> and mmhub_v1_0_program_invalidation().
>
> Change-Id: I9b9b44f17ac2a6ff0c9c78f91885665da75543d0
> Signed-off-by: Yong Zhao 
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c | 17 -
>  1 file changed, 17 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c
> index 60b5f56c..3ade5d5 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c
> @@ -60,11 +60,6 @@
>  #define mmMMHUB_VM_INVALIDATE_ENG16_ACK  0x0705
>  #define mmMMHUB_VM_INVALIDATE_ENG16_ACK_BASE_IDX 0
>  
> -#define mmMMHUB_VM_INVALIDATE_ENG16_ADDR_RANGE_LO32  0x0727
> -#define mmMMHUB_VM_INVALIDATE_ENG16_ADDR_RANGE_LO32_BASE_IDX 0
> -#define mmMMHUB_VM_INVALIDATE_ENG16_ADDR_RANGE_HI32  0x0728
> -#define mmMMHUB_VM_INVALIDATE_ENG16_ADDR_RANGE_HI32_BASE_IDX 0
> -
>  #define V9_PIPE_PER_MEC  (4)
>  #define V9_QUEUES_PER_PIPE_MEC   (8)
>  
> @@ -772,18 +767,6 @@ static void write_vmid_invalidate_request(struct kgd_dev 
> *kgd, uint8_t vmid)
>* TODO 2: support range-based invalidation, requires kfg2kgd
>* interface change
>*/
> - WREG32(SOC15_REG_OFFSET(GC, 0, mmVM_INVALIDATE_ENG16_ADDR_RANGE_LO32),
> - 0x);
> - WREG32(SOC15_REG_OFFSET(GC, 0, mmVM_INVALIDATE_ENG16_ADDR_RANGE_HI32),
> - 0x001f);
> -
> - WREG32(SOC15_REG_OFFSET(MMHUB, 0,
> - mmMMHUB_VM_INVALIDATE_ENG16_ADDR_RANGE_LO32),
> - 0x);
> - WREG32(SOC15_REG_OFFSET(MMHUB, 0,
> - mmMMHUB_VM_INVALIDATE_ENG16_ADDR_RANGE_HI32),
> - 0x001f);
> -
>   WREG32(SOC15_REG_OFFSET(GC, 0, mmVM_INVALIDATE_ENG16_REQ), req);
>  
>   WREG32(SOC15_REG_OFFSET(MMHUB, 0, mmMMHUB_VM_INVALIDATE_ENG16_REQ),
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 1/2] drm/amdgpu: Reorganize *_flush_gpu_tlb() for kfd to use

[Kuehling, Felix]

It occurred to me that the flush_type is a hardware-specific value, but
you're using it in a hardware-abstracted interface. If the meaning of
the flush type values changes in future HW-generations, we'll need to
define an abstract enum that gets translated to the respective HW values
in the HW-specific code.

Anyway, this works for now.

One more cosmetic comment inline ...

Other than that the series is Reviewed-by: Felix Kuehling


Regards,
  Felix

On 2018-10-23 2:15 p.m., Zhao, Yong wrote:
> Add a flush_type parameter to that series of functions.
>
> Change-Id: I3dcd71955297c53b181f82e7078981230c642c01
> Signed-off-by: Yong Zhao 
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c |  4 ++--
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h  |  4 ++--
>  drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c|  5 +++--
>  drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c|  5 +++--
>  drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c|  4 ++--
>  drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c| 19 ++-
>  6 files changed, 22 insertions(+), 19 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c
> index 11fea28..9a212aa 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c
> @@ -248,7 +248,7 @@ int amdgpu_gart_unbind(struct amdgpu_device *adev, 
> uint64_t offset,
>   }
>   mb();
>   amdgpu_asic_flush_hdp(adev, NULL);
> - amdgpu_gmc_flush_gpu_tlb(adev, 0);
> + amdgpu_gmc_flush_gpu_tlb(adev, 0, 0);
>   return 0;
>  }
>  
> @@ -331,7 +331,7 @@ int amdgpu_gart_bind(struct amdgpu_device *adev, uint64_t 
> offset,
>  
>   mb();
>   amdgpu_asic_flush_hdp(adev, NULL);
> - amdgpu_gmc_flush_gpu_tlb(adev, 0);
> + amdgpu_gmc_flush_gpu_tlb(adev, 0, 0);
>   return 0;
>  }
>  
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> index 6fa7ef4..4c5f18c 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> @@ -64,7 +64,7 @@ struct amdgpu_vmhub {
>  struct amdgpu_gmc_funcs {
>   /* flush the vm tlb via mmio */
>   void (*flush_gpu_tlb)(struct amdgpu_device *adev,
> -   uint32_t vmid);
> +   uint32_t vmid, uint32_t flush_type);
>   /* flush the vm tlb via ring */
>   uint64_t (*emit_flush_gpu_tlb)(struct amdgpu_ring *ring, unsigned vmid,
>  uint64_t pd_addr);
> @@ -151,7 +151,7 @@ struct amdgpu_gmc {
>   struct amdgpu_xgmi xgmi;
>  };
>  
> -#define amdgpu_gmc_flush_gpu_tlb(adev, vmid) 
> (adev)->gmc.gmc_funcs->flush_gpu_tlb((adev), (vmid))
> +#define amdgpu_gmc_flush_gpu_tlb(adev, vmid, type) 
> (adev)->gmc.gmc_funcs->flush_gpu_tlb((adev), (vmid), (type))
>  #define amdgpu_gmc_emit_flush_gpu_tlb(r, vmid, addr) 
> (r)->adev->gmc.gmc_funcs->emit_flush_gpu_tlb((r), (vmid), (addr))
>  #define amdgpu_gmc_emit_pasid_mapping(r, vmid, pasid) 
> (r)->adev->gmc.gmc_funcs->emit_pasid_mapping((r), (vmid), (pasid))
>  #define amdgpu_gmc_set_pte_pde(adev, pt, idx, addr, flags) 
> (adev)->gmc.gmc_funcs->set_pte_pde((adev), (pt), (idx), (addr), (flags))
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c 
> b/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
> index e1c2b4e..2821d1d 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
> @@ -358,7 +358,8 @@ static int gmc_v6_0_mc_init(struct amdgpu_device *adev)
>   return 0;
>  }
>  
> -static void gmc_v6_0_flush_gpu_tlb(struct amdgpu_device *adev, uint32_t vmid)
> +static void gmc_v6_0_flush_gpu_tlb(struct amdgpu_device *adev,
> + uint32_t vmid, uint32_t flush_type)
>  {
>   WREG32(mmVM_INVALIDATE_REQUEST, 1 << vmid);
>  }
> @@ -580,7 +581,7 @@ static int gmc_v6_0_gart_enable(struct amdgpu_device 
> *adev)
>   else
>   gmc_v6_0_set_fault_enable_default(adev, true);
>  
> - gmc_v6_0_flush_gpu_tlb(adev, 0);
> + gmc_v6_0_flush_gpu_tlb(adev, 0, 0);
>   dev_info(adev->dev, "PCIE GART of %uM enabled (table at 0x%016llX).\n",
>(unsigned)(adev->gmc.gart_size >> 20),
>(unsigned long long)table_addr);
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c 
> b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
> index 910c4ce..761dcfb 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
> @@ -430,7 +430,8 @@ static int gmc_v7_0_mc_init(struct amdgpu_device *adev)
>   *
>   * Flush the TLB for the requested page table (CIK).
>   */
> -static void gmc_v7_0_flush_gpu_tlb(struct amdgpu_device *adev, uint32_t vmid)
> +static void gmc_v7_0_flush_gpu_tlb(struct amdgpu_device *adev,
> + uint32_t vmid, uint32_t flush_type)
>  {
>   /* bits 0-15 are the VM contexts0-15 */
>   WREG32(mmVM_INVALIDATE_REQUEST, 1 << vmid);
> @@ -698,7 +699,7 @@ static int gmc_v7_0_gart_enable(struct amdgpu_

Re: [PATCH 2/2] drm/amdkfd: page_table_base already have the flags needed

[Kuehling, Felix]

The series is Reviewed-by: Felix Kuehling 


On 2018-10-23 1:00 p.m., Zhao, Yong wrote:
>
> How about those two patches?
>
>
> Yong
>
> 
> *From:* Zhao, Yong
> *Sent:* Monday, October 22, 2018 2:33:26 PM
> *To:* amd-gfx@lists.freedesktop.org; brahma_sw_dev
> *Cc:* Zhao, Yong
> *Subject:* [PATCH 2/2] drm/amdkfd: page_table_base already have the
> flags needed
>  
> The flags are added when calling amdgpu_gmc_pd_addr().
>
> Change-Id: Idd85b1ac35d3d100154df8229ea20721d9a7045c
> Signed-off-by: Yong Zhao 
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c | 5 ++---
>  drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 1 +
>  2 files changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c
> index 54c3690..60b5f56c 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c
> @@ -978,7 +978,6 @@ static void set_vm_context_page_table_base(struct
> kgd_dev *kgd, uint32_t vmid,
>  uint64_t page_table_base)
>  {
>  struct amdgpu_device *adev = get_amdgpu_device(kgd);
> -   uint64_t base = page_table_base | AMDGPU_PTE_VALID;
>  
>  if (!amdgpu_amdkfd_is_kfd_vmid(adev, vmid)) {
>  pr_err("trying to set page table base for wrong VMID
> %u\n",
> @@ -990,7 +989,7 @@ static void set_vm_context_page_table_base(struct
> kgd_dev *kgd, uint32_t vmid,
>   * now, all processes share the same address space size, like
>   * on GFX8 and older.
>   */
> -   mmhub_v1_0_setup_vm_pt_regs(adev, vmid, base);
> +   mmhub_v1_0_setup_vm_pt_regs(adev, vmid, page_table_base);
>  
> -   gfxhub_v1_0_setup_vm_pt_regs(adev, vmid, base);
> +   gfxhub_v1_0_setup_vm_pt_regs(adev, vmid, page_table_base);
>  }
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
> b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
> index 53ff86d..dec8e64 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
> @@ -507,6 +507,7 @@ struct qcm_process_device {
>   * All the memory management data should be here too
>   */
>  uint64_t gds_context_area;
> +   /* Contains page table flags such as AMDGPU_PTE_VALID since
> gfx9 */
>  uint64_t page_table_base;
>  uint32_t sh_mem_config;
>  uint32_t sh_mem_bases;
> -- 
> 2.7.4
>
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdgpu: fix VM leaf walking

[Kuehling, Felix]

On 2018-10-25 10:38 a.m., Christian König wrote:
> Make sure we don't try to go down further after the leave walk already
> ended. This fixes a crash with a new VM test.
>
> Signed-off-by: Christian König 

Reviewed-by: Felix Kuehling 

Regards,
  Felix

> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index db0cbf8d219d..352b30409060 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -542,7 +542,8 @@ static void amdgpu_vm_pt_next_leaf(struct amdgpu_device 
> *adev,
>  struct amdgpu_vm_pt_cursor *cursor)
>  {
>   amdgpu_vm_pt_next(adev, cursor);
> - while (amdgpu_vm_pt_descendant(adev, cursor));
> + if (cursor->pfn != ~0ll)
> + while (amdgpu_vm_pt_descendant(adev, cursor));
>  }
>  
>  /**
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdgpu/amdkfd: clean up mmhub and gfxhub includes

[Kuehling, Felix]

On 2018-10-25 2:27 p.m., Alex Deucher wrote:
> On Mon, Oct 22, 2018 at 6:25 PM Alex Deucher  wrote:
>> Use the appropriate mmhub and gfxhub headers rather than adding
>> them to the gmc9 header.
>>
>> Signed-off-by: Alex Deucher 
> Ping?

Reviewed-by: Felix Kuehling 


>
> Alex
>
>> ---
>>  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c | 3 ++-
>>  drivers/gpu/drm/amd/amdgpu/gfxhub_v1_0.h  | 2 ++
>>  drivers/gpu/drm/amd/amdgpu/gmc_v9_0.h | 6 --
>>  drivers/gpu/drm/amd/amdgpu/mmhub_v1_0.h   | 2 ++
>>  4 files changed, 6 insertions(+), 7 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c 
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c
>> index 54c369091f6c..02d1d363931b 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c
>> @@ -46,7 +46,8 @@
>>  #include "v9_structs.h"
>>  #include "soc15.h"
>>  #include "soc15d.h"
>> -#include "gmc_v9_0.h"
>> +#include "mmhub_v1_0.h"
>> +#include "gfxhub_v1_0.h"
>>
>>  /* HACK: MMHUB and GC both have VM-related register with the same
>>   * names but different offsets. Define the MMHUB register we need here
>> diff --git a/drivers/gpu/drm/amd/amdgpu/gfxhub_v1_0.h 
>> b/drivers/gpu/drm/amd/amdgpu/gfxhub_v1_0.h
>> index 206e29cad753..92d3a70cd9b1 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/gfxhub_v1_0.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/gfxhub_v1_0.h
>> @@ -30,5 +30,7 @@ void gfxhub_v1_0_set_fault_enable_default(struct 
>> amdgpu_device *adev,
>>   bool value);
>>  void gfxhub_v1_0_init(struct amdgpu_device *adev);
>>  u64 gfxhub_v1_0_get_mc_fb_offset(struct amdgpu_device *adev);
>> +void gfxhub_v1_0_setup_vm_pt_regs(struct amdgpu_device *adev, uint32_t vmid,
>> +   uint64_t page_table_base);
>>
>>  #endif
>> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.h 
>> b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.h
>> index 1fd178a65e66..b030ca5ea107 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.h
>> @@ -27,10 +27,4 @@
>>  extern const struct amd_ip_funcs gmc_v9_0_ip_funcs;
>>  extern const struct amdgpu_ip_block_version gmc_v9_0_ip_block;
>>
>> -/* amdgpu_amdkfd*.c */
>> -void gfxhub_v1_0_setup_vm_pt_regs(struct amdgpu_device *adev, uint32_t vmid,
>> -   uint64_t page_table_base);
>> -void mmhub_v1_0_setup_vm_pt_regs(struct amdgpu_device *adev, uint32_t vmid,
>> -   uint64_t page_table_base);
>> -
>>  #endif
>> diff --git a/drivers/gpu/drm/amd/amdgpu/mmhub_v1_0.h 
>> b/drivers/gpu/drm/amd/amdgpu/mmhub_v1_0.h
>> index bef3d0c0c117..0de0fdf98c00 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/mmhub_v1_0.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/mmhub_v1_0.h
>> @@ -34,5 +34,7 @@ int mmhub_v1_0_set_clockgating(struct amdgpu_device *adev,
>>  void mmhub_v1_0_get_clockgating(struct amdgpu_device *adev, u32 *flags);
>>  void mmhub_v1_0_update_power_gating(struct amdgpu_device *adev,
>>  bool enable);
>> +void mmhub_v1_0_setup_vm_pt_regs(struct amdgpu_device *adev, uint32_t vmid,
>> +   uint64_t page_table_base);
>>
>>  #endif
>> --
>> 2.13.6
>>
> ___
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdkfd: fix interrupt spin lock

[Kuehling, Felix]

On 2018-11-02 9:48 a.m., Christian König wrote:
> Vega10 has multiple interrupt rings,

I don't think I've seen your code that implements multiple interrupt
rings. So it's a bit hard to comment. As I understand it, the only way
this could happen is, if the two interrupt rings are handled by
different CPU cores. Alternatively, you may end up calling
kgd2kfd_interrupt outside of interrupt context (e.g. in a kernel
thread), which would invalidate an implicit assumption in
kgd2kfd_interrupt. Can you clarify what's happening here?

Anyway, this change shouldn't cause any other problems so feel free to
add my Acked-by.

Regards,
  Felix


>  so this can be called from multiple
> calles at the same time resulting in:
>
> [   71.779334] 
> [   71.779406] WARNING: inconsistent lock state
> [   71.779478] 4.19.0-rc1+ #44 Tainted: GW
> [   71.779565] 
> [   71.779637] inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage.
> [   71.779740] kworker/6:1/120 [HC0[0]:SC0[0]:HE1:SE1] takes:
> [   71.779832] ad761971 (&(&kfd->interrupt_lock)->rlock){?...},
> at: kgd2kfd_interrupt+0x75/0x100 [amdgpu]
> [   71.780058] {IN-HARDIRQ-W} state was registered at:
> [   71.780115]   _raw_spin_lock+0x2c/0x40
> [   71.780180]   kgd2kfd_interrupt+0x75/0x100 [amdgpu]
> [   71.780248]   amdgpu_irq_callback+0x6c/0x150 [amdgpu]
> [   71.780315]   amdgpu_ih_process+0x88/0x100 [amdgpu]
> [   71.780380]   amdgpu_irq_handler+0x20/0x40 [amdgpu]
> [   71.780409]   __handle_irq_event_percpu+0x49/0x2a0
> [   71.780436]   handle_irq_event_percpu+0x30/0x70
> [   71.780461]   handle_irq_event+0x37/0x60
> [   71.780484]   handle_edge_irq+0x83/0x1b0
> [   71.780506]   handle_irq+0x1f/0x30
> [   71.780526]   do_IRQ+0x53/0x110
> [   71.780544]   ret_from_intr+0x0/0x22
> [   71.780566]   cpuidle_enter_state+0xaa/0x330
> [   71.780591]   do_idle+0x203/0x280
> [   71.780610]   cpu_startup_entry+0x6f/0x80
> [   71.780634]   start_secondary+0x1b0/0x200
> [   71.780657]   secondary_startup_64+0xa4/0xb0
>
> Fix this by always using irq save spin locks.
>
> Signed-off-by: Christian König 
> ---
>  drivers/gpu/drm/amd/amdkfd/kfd_device.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c 
> b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
> index d7e0f41f8edc..c004647c8cb4 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
> @@ -682,6 +682,7 @@ void kgd2kfd_interrupt(struct kfd_dev *kfd, const void 
> *ih_ring_entry)
>  {
>   uint32_t patched_ihre[KFD_MAX_RING_ENTRY_SIZE];
>   bool is_patched = false;
> + unsigned long flags;
>  
>   if (!kfd->init_complete)
>   return;
> @@ -691,7 +692,7 @@ void kgd2kfd_interrupt(struct kfd_dev *kfd, const void 
> *ih_ring_entry)
>   return;
>   }
>  
> - spin_lock(&kfd->interrupt_lock);
> + spin_lock_irqsave(&kfd->interrupt_lock, flags);
>  
>   if (kfd->interrupts_active
>   && interrupt_is_wanted(kfd, ih_ring_entry,
> @@ -700,7 +701,7 @@ void kgd2kfd_interrupt(struct kfd_dev *kfd, const void 
> *ih_ring_entry)
>is_patched ? patched_ihre : ih_ring_entry))
>   queue_work(kfd->ih_wq, &kfd->interrupt_work);
>  
> - spin_unlock(&kfd->interrupt_lock);
> + spin_unlock_irqrestore(&kfd->interrupt_lock, flags);
>  }
>  
>  int kgd2kfd_quiesce_mm(struct mm_struct *mm)
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdkfd: fix interrupt spin lock

[Kuehling, Felix]

On 2018-11-04 2:20 p.m., Christian König wrote:
> Am 02.11.18 um 19:59 schrieb Kuehling, Felix:
>> On 2018-11-02 9:48 a.m., Christian König wrote:
>>> Vega10 has multiple interrupt rings,
>> I don't think I've seen your code that implements multiple interrupt
>> rings. So it's a bit hard to comment.
>
> It's already a while ago, but you actually reviewed it :)
>
>> As I understand it, the only way
>> this could happen is, if the two interrupt rings are handled by
>> different CPU cores. Alternatively, you may end up calling
>> kgd2kfd_interrupt outside of interrupt context (e.g. in a kernel
>> thread), which would invalidate an implicit assumption in
>> kgd2kfd_interrupt. Can you clarify what's happening here?
>
> Currently the later scenario, I've offloaded ring processing on IH
> ring 1 & 2 to a work item.
>
> But the IH is in theory capable of raising IRQ on different CPU cores
> as well, e.g. first halve of ring is processed and acknowledged on
> CPU1, new interrupt for next entries is raised on CPU2 before CPU1
> finishes executing the handler.
>
> We currently explicitly prevent that in the amdgpu IH code, but that
> is something I would really really like to get rid of.
>
> Are there any other cases where this is assumed in the
> kgd2kfd_interrupt code?

It should be fine. We have only a small interrupt handler that puts
interesting interrupts into a kfifo. That is protected by the
kfd->interrupt_lock spinlock. So all that code is running in atomic
context either way. If there are multiple interrupt threads, the
spinlock will serialize them.

The interesting work (fault processing and event handling) happens in
the worker thread that processes the kfifo.

Regards,
  Felix


>
> Christian.
>
>>
>> Anyway, this change shouldn't cause any other problems so feel free to
>> add my Acked-by.
>>
>> Regards,
>>    Felix
>>
>>
>>>   so this can be called from multiple
>>> calles at the same time resulting in:
>>>
>>> [   71.779334] 
>>> [   71.779406] WARNING: inconsistent lock state
>>> [   71.779478] 4.19.0-rc1+ #44 Tainted: G    W
>>> [   71.779565] 
>>> [   71.779637] inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage.
>>> [   71.779740] kworker/6:1/120 [HC0[0]:SC0[0]:HE1:SE1] takes:
>>> [   71.779832] ad761971 (&(&kfd->interrupt_lock)->rlock){?...},
>>> at: kgd2kfd_interrupt+0x75/0x100 [amdgpu]
>>> [   71.780058] {IN-HARDIRQ-W} state was registered at:
>>> [   71.780115]   _raw_spin_lock+0x2c/0x40
>>> [   71.780180]   kgd2kfd_interrupt+0x75/0x100 [amdgpu]
>>> [   71.780248]   amdgpu_irq_callback+0x6c/0x150 [amdgpu]
>>> [   71.780315]   amdgpu_ih_process+0x88/0x100 [amdgpu]
>>> [   71.780380]   amdgpu_irq_handler+0x20/0x40 [amdgpu]
>>> [   71.780409]   __handle_irq_event_percpu+0x49/0x2a0
>>> [   71.780436]   handle_irq_event_percpu+0x30/0x70
>>> [   71.780461]   handle_irq_event+0x37/0x60
>>> [   71.780484]   handle_edge_irq+0x83/0x1b0
>>> [   71.780506]   handle_irq+0x1f/0x30
>>> [   71.780526]   do_IRQ+0x53/0x110
>>> [   71.780544]   ret_from_intr+0x0/0x22
>>> [   71.780566]   cpuidle_enter_state+0xaa/0x330
>>> [   71.780591]   do_idle+0x203/0x280
>>> [   71.780610]   cpu_startup_entry+0x6f/0x80
>>> [   71.780634]   start_secondary+0x1b0/0x200
>>> [   71.780657]   secondary_startup_64+0xa4/0xb0
>>>
>>> Fix this by always using irq save spin locks.
>>>
>>> Signed-off-by: Christian König 
>>> ---
>>>   drivers/gpu/drm/amd/amdkfd/kfd_device.c | 5 +++--
>>>   1 file changed, 3 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
>>> b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
>>> index d7e0f41f8edc..c004647c8cb4 100644
>>> --- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
>>> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
>>> @@ -682,6 +682,7 @@ void kgd2kfd_interrupt(struct kfd_dev *kfd,
>>> const void *ih_ring_entry)
>>>   {
>>>   uint32_t patched_ihre[KFD_MAX_RING_ENTRY_SIZE];
>>>   bool is_patched = false;
>>> +    unsigned long flags;
>>>     if (!kfd->init_complete)
>>>   return;
>>> @@ -691,7 +692,7 @@ void kgd2kfd_interrupt(struct kfd_dev *kfd,
>>> const void *ih_ring_entry)
>>>   return;
>>>   }
>>>   -    spin_lock(&kfd->interrupt_lock);
>>> +    spin_lock_irqsave(&kfd->interrupt_lock, flags);
>>>     if (kfd->interrupts_active
>>>   && interrupt_is_wanted(kfd, ih_ring_entry,
>>> @@ -700,7 +701,7 @@ void kgd2kfd_interrupt(struct kfd_dev *kfd,
>>> const void *ih_ring_entry)
>>>    is_patched ? patched_ihre : ih_ring_entry))
>>>   queue_work(kfd->ih_wq, &kfd->interrupt_work);
>>>   -    spin_unlock(&kfd->interrupt_lock);
>>> +    spin_unlock_irqrestore(&kfd->interrupt_lock, flags);
>>>   }
>>>     int kgd2kfd_quiesce_mm(struct mm_struct *mm)
>
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 0/9] KFD upstreaming Nov 2018, part 1

[Kuehling, Felix]

These are some recent patches that are easy to upstream (part 1). For
part 2 (hopefully still this month) I'll need to advance the merging
of KFD into amdgpu a little further to avoid upstreaming duplicated
data structures that no longer need to be duplicated.

Eric Huang (1):
  drm/amdkfd: change system memory overcommit limit

Felix Kuehling (2):
  drm/amdkfd: Fix and simplify sync object handling for KFD
  drm/amdgpu: Fix KFD doorbell SG BO mapping

Gang Ba (1):
  drm/amdkfd: Added Vega12 and Polaris12 for KFD.

Harish Kasiviswanathan (2):
  drm/amdgpu: Remove explicit wait after VM validate
  drm/amdgpu: KFD Restore process: Optimize waiting

Yong Zhao (3):
  drm/amdkfd: Replace mqd with mqd_mgr as the variable name for
mqd_manager
  drm/amdkfd: Adjust the debug message in KFD ISR
  drm/amdkfd: Workaround PASID missing in gfx9 interrupt payload under
non HWS

 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c |   2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c   | 171 -
 drivers/gpu/drm/amd/amdkfd/kfd_crat.c  |   6 +
 drivers/gpu/drm/amd/amdkfd/kfd_device.c|  45 ++
 .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.c  |  14 +-
 drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c   |   2 +
 drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c|  35 -
 drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.c  |   2 +
 drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager.c   |   2 +
 drivers/gpu/drm/amd/amdkfd/kfd_packet_manager.c|   2 +
 drivers/gpu/drm/amd/amdkfd/kfd_topology.c  |   2 +
 11 files changed, 194 insertions(+), 89 deletions(-)

-- 
2.7.4

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 1/9] drm/amdkfd: Replace mqd with mqd_mgr as the variable name for mqd_manager

[Kuehling, Felix]

From: Yong Zhao 

This will make reading code much easier. This fixes a few spots missed in a
previous commit with the same title.

Signed-off-by: Yong Zhao 
Reviewed-by: Felix Kuehling 
Signed-off-by: Felix Kuehling 
---
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c | 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c 
b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
index fb9d66e..d38efbb 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
@@ -1547,7 +1547,7 @@ static int get_wave_state(struct device_queue_manager 
*dqm,
  u32 *ctl_stack_used_size,
  u32 *save_area_used_size)
 {
-   struct mqd_manager *mqd;
+   struct mqd_manager *mqd_mgr;
int r;
 
dqm_lock(dqm);
@@ -1558,19 +1558,19 @@ static int get_wave_state(struct device_queue_manager 
*dqm,
goto dqm_unlock;
}
 
-   mqd = dqm->ops.get_mqd_manager(dqm, KFD_MQD_TYPE_COMPUTE);
-   if (!mqd) {
+   mqd_mgr = dqm->ops.get_mqd_manager(dqm, KFD_MQD_TYPE_COMPUTE);
+   if (!mqd_mgr) {
r = -ENOMEM;
goto dqm_unlock;
}
 
-   if (!mqd->get_wave_state) {
+   if (!mqd_mgr->get_wave_state) {
r = -EINVAL;
goto dqm_unlock;
}
 
-   r = mqd->get_wave_state(mqd, q->mqd, ctl_stack, ctl_stack_used_size,
-   save_area_used_size);
+   r = mqd_mgr->get_wave_state(mqd_mgr, q->mqd, ctl_stack,
+   ctl_stack_used_size, save_area_used_size);
 
 dqm_unlock:
dqm_unlock(dqm);
-- 
2.7.4

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 7/9] drm/amdkfd: Fix and simplify sync object handling for KFD

[Kuehling, Felix]

The adev parameter in amdgpu_sync_fence and amdgpu_sync_resv is only
needed for updating sync->last_vm_update. This breaks if different
adevs are passed to calls for the same sync object.

Always pass NULL for calls from KFD because sync objects used for
KFD don't belong to any particular device, and KFD doesn't need the
sync->last_vm_update fence.

This fixes kernel log warnings on multi-GPU systems after recent
changes in amdgpu_amdkfd_gpuvm_restore_process_bos.

Signed-off-by: Felix Kuehling 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 28 +---
 1 file changed, 5 insertions(+), 23 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
index d005371..572ac5f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
@@ -395,23 +395,6 @@ static int vm_validate_pt_pd_bos(struct amdgpu_vm *vm)
return 0;
 }
 
-static int sync_vm_fence(struct amdgpu_device *adev, struct amdgpu_sync *sync,
-struct dma_fence *f)
-{
-   int ret = amdgpu_sync_fence(adev, sync, f, false);
-
-   /* Sync objects can't handle multiple GPUs (contexts) updating
-* sync->last_vm_update. Fortunately we don't need it for
-* KFD's purposes, so we can just drop that fence.
-*/
-   if (sync->last_vm_update) {
-   dma_fence_put(sync->last_vm_update);
-   sync->last_vm_update = NULL;
-   }
-
-   return ret;
-}
-
 static int vm_update_pds(struct amdgpu_vm *vm, struct amdgpu_sync *sync)
 {
struct amdgpu_bo *pd = vm->root.base.bo;
@@ -422,7 +405,7 @@ static int vm_update_pds(struct amdgpu_vm *vm, struct 
amdgpu_sync *sync)
if (ret)
return ret;
 
-   return sync_vm_fence(adev, sync, vm->last_update);
+   return amdgpu_sync_fence(NULL, sync, vm->last_update, false);
 }
 
 /* add_bo_to_vm - Add a BO to a VM
@@ -826,7 +809,7 @@ static int unmap_bo_from_gpuvm(struct amdgpu_device *adev,
/* Add the eviction fence back */
amdgpu_bo_fence(pd, &vm->process_info->eviction_fence->base, true);
 
-   sync_vm_fence(adev, sync, bo_va->last_pt_update);
+   amdgpu_sync_fence(NULL, sync, bo_va->last_pt_update, false);
 
return 0;
 }
@@ -851,7 +834,7 @@ static int update_gpuvm_pte(struct amdgpu_device *adev,
return ret;
}
 
-   return sync_vm_fence(adev, sync, bo_va->last_pt_update);
+   return amdgpu_sync_fence(NULL, sync, bo_va->last_pt_update, false);
 }
 
 static int map_bo_to_gpuvm(struct amdgpu_device *adev,
@@ -911,7 +894,7 @@ static int process_sync_pds_resv(struct amdkfd_process_info 
*process_info,
vm_list_node) {
struct amdgpu_bo *pd = peer_vm->root.base.bo;
 
-   ret = amdgpu_sync_resv(amdgpu_ttm_adev(pd->tbo.bdev),
+   ret = amdgpu_sync_resv(NULL,
sync, pd->tbo.resv,
AMDGPU_FENCE_OWNER_UNDEFINED, false);
if (ret)
@@ -2084,8 +2067,7 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, 
struct dma_fence **ef)
pr_debug("Memory eviction: Validate BOs failed. Try 
again\n");
goto validate_map_fail;
}
-   ret = amdgpu_sync_fence(amdgpu_ttm_adev(bo->tbo.bdev),
-   &sync_obj, bo->tbo.moving, false);
+   ret = amdgpu_sync_fence(NULL, &sync_obj, bo->tbo.moving, false);
if (ret) {
pr_debug("Memory eviction: Sync BO fence failed. Try 
again\n");
goto validate_map_fail;
-- 
2.7.4

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 5/9] drm/amdgpu: Remove explicit wait after VM validate

[Kuehling, Felix]

From: Harish Kasiviswanathan 

PD or PT might have to be moved during validation and this move has to be
completed before updating it. If page table updates are done using SDMA
then this serializing is done by SDMA command submission.

And if PD/PT updates are done by CPU, then explicit waiting for PD/PT
updates are done in amdgpu VM amdgpu_vm_wait_pd function.

Sync to PD BO moving fence to handle corner case where none of the PTs
are updated but PD is evicted.

Signed-off-by: Harish Kasiviswanathan 
Reviewed-by: Felix Kuehling 
Signed-off-by: Felix Kuehling 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 31 ++--
 1 file changed, 24 insertions(+), 7 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
index 9a1b2b2..e124d2d 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
@@ -901,6 +901,26 @@ static int process_validate_vms(struct amdkfd_process_info 
*process_info)
return 0;
 }
 
+static int process_sync_pds_resv(struct amdkfd_process_info *process_info,
+struct amdgpu_sync *sync)
+{
+   struct amdgpu_vm *peer_vm;
+   int ret;
+
+   list_for_each_entry(peer_vm, &process_info->vm_list_head,
+   vm_list_node) {
+   struct amdgpu_bo *pd = peer_vm->root.base.bo;
+
+   ret = amdgpu_sync_resv(amdgpu_ttm_adev(pd->tbo.bdev),
+   sync, pd->tbo.resv,
+   AMDGPU_FENCE_OWNER_UNDEFINED, false);
+   if (ret)
+   return ret;
+   }
+
+   return 0;
+}
+
 static int process_update_pds(struct amdkfd_process_info *process_info,
  struct amdgpu_sync *sync)
 {
@@ -2045,13 +2065,10 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, 
struct dma_fence **ef)
if (ret)
goto validate_map_fail;
 
-   /* Wait for PD/PTs validate to finish */
-   /* FIXME: I think this isn't needed */
-   list_for_each_entry(peer_vm, &process_info->vm_list_head,
-   vm_list_node) {
-   struct amdgpu_bo *bo = peer_vm->root.base.bo;
-
-   ttm_bo_wait(&bo->tbo, false, false);
+   ret = process_sync_pds_resv(process_info, &sync_obj);
+   if (ret) {
+   pr_debug("Memory eviction: Failed to sync to PD BO moving 
fence. Try again\n");
+   goto validate_map_fail;
}
 
/* Validate BOs and map them to GPUVM (update VM page tables). */
-- 
2.7.4

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 6/9] drm/amdgpu: KFD Restore process: Optimize waiting

[Kuehling, Felix]

From: Harish Kasiviswanathan 

Instead of waiting for each KFD BO after validation just wait for the
last BO moving fence.

Signed-off-by: Harish Kasiviswanathan 
Reviewed-by: Felix Kuehling 
Signed-off-by: Felix Kuehling 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 13 -
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
index e124d2d..d005371 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
@@ -2084,7 +2084,12 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, 
struct dma_fence **ef)
pr_debug("Memory eviction: Validate BOs failed. Try 
again\n");
goto validate_map_fail;
}
-
+   ret = amdgpu_sync_fence(amdgpu_ttm_adev(bo->tbo.bdev),
+   &sync_obj, bo->tbo.moving, false);
+   if (ret) {
+   pr_debug("Memory eviction: Sync BO fence failed. Try 
again\n");
+   goto validate_map_fail;
+   }
list_for_each_entry(bo_va_entry, &mem->bo_va_list,
bo_list) {
ret = update_gpuvm_pte((struct amdgpu_device *)
@@ -2105,6 +2110,7 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, 
struct dma_fence **ef)
goto validate_map_fail;
}
 
+   /* Wait for validate and PT updates to finish */
amdgpu_sync_wait(&sync_obj, false);
 
/* Release old eviction fence and create new one, because fence only
@@ -2123,10 +2129,7 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, 
struct dma_fence **ef)
process_info->eviction_fence = new_fence;
*ef = dma_fence_get(&new_fence->base);
 
-   /* Wait for validate to finish and attach new eviction fence */
-   list_for_each_entry(mem, &process_info->kfd_bo_list,
-   validate_list.head)
-   ttm_bo_wait(&mem->bo->tbo, false, false);
+   /* Attach new eviction fence to all BOs */
list_for_each_entry(mem, &process_info->kfd_bo_list,
validate_list.head)
amdgpu_bo_fence(mem->bo,
-- 
2.7.4

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 2/9] drm/amdkfd: Added Vega12 and Polaris12 for KFD.

[Kuehling, Felix]

From: Gang Ba 

Add Vega12 and Polaris12 device info and device IDs to KFD.

Signed-off-by: Gang Ba 
Reviewed-by: Felix Kuehling 
Signed-off-by: Felix Kuehling 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c |  2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c   |  3 +-
 drivers/gpu/drm/amd/amdkfd/kfd_crat.c  |  6 +++
 drivers/gpu/drm/amd/amdkfd/kfd_device.c| 45 ++
 .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.c  |  2 +
 drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c   |  2 +
 drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.c  |  2 +
 drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager.c   |  2 +
 drivers/gpu/drm/amd/amdkfd/kfd_packet_manager.c|  2 +
 drivers/gpu/drm/amd/amdkfd/kfd_topology.c  |  2 +
 10 files changed, 67 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
index 60f9a87..a9c7597 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
@@ -73,9 +73,11 @@ void amdgpu_amdkfd_device_probe(struct amdgpu_device *adev)
case CHIP_FIJI:
case CHIP_POLARIS10:
case CHIP_POLARIS11:
+   case CHIP_POLARIS12:
kfd2kgd = amdgpu_amdkfd_gfx_8_0_get_functions();
break;
case CHIP_VEGA10:
+   case CHIP_VEGA12:
case CHIP_VEGA20:
case CHIP_RAVEN:
kfd2kgd = amdgpu_amdkfd_gfx_9_0_get_functions();
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
index df0a059..9a1b2b2 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
@@ -1199,7 +1199,8 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(
byte_align = (adev->family == AMDGPU_FAMILY_VI &&
adev->asic_type != CHIP_FIJI &&
adev->asic_type != CHIP_POLARIS10 &&
-   adev->asic_type != CHIP_POLARIS11) ?
+   adev->asic_type != CHIP_POLARIS11 &&
+   adev->asic_type != CHIP_POLARIS12) ?
VI_BO_SIZE_ALIGN : 1;
 
mapping_flags = AMDGPU_VM_PAGE_READABLE;
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c 
b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
index 3783d12..c02adbb 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
@@ -133,6 +133,7 @@ static struct kfd_gpu_cache_info carrizo_cache_info[] = {
 #define fiji_cache_info  carrizo_cache_info
 #define polaris10_cache_info carrizo_cache_info
 #define polaris11_cache_info carrizo_cache_info
+#define polaris12_cache_info carrizo_cache_info
 /* TODO - check & update Vega10 cache details */
 #define vega10_cache_info carrizo_cache_info
 #define raven_cache_info carrizo_cache_info
@@ -647,7 +648,12 @@ static int kfd_fill_gpu_cache_info(struct kfd_dev *kdev,
pcache_info = polaris11_cache_info;
num_of_cache_types = ARRAY_SIZE(polaris11_cache_info);
break;
+   case CHIP_POLARIS12:
+   pcache_info = polaris12_cache_info;
+   num_of_cache_types = ARRAY_SIZE(polaris12_cache_info);
+   break;
case CHIP_VEGA10:
+   case CHIP_VEGA12:
case CHIP_VEGA20:
pcache_info = vega10_cache_info;
num_of_cache_types = ARRAY_SIZE(vega10_cache_info);
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c 
b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
index c004647..9ed14a1 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
@@ -205,6 +205,22 @@ static const struct kfd_device_info polaris11_device_info 
= {
.num_sdma_queues_per_engine = 2,
 };
 
+static const struct kfd_device_info polaris12_device_info = {
+   .asic_family = CHIP_POLARIS12,
+   .max_pasid_bits = 16,
+   .max_no_of_hqd  = 24,
+   .doorbell_size  = 4,
+   .ih_ring_entry_size = 4 * sizeof(uint32_t),
+   .event_interrupt_class = &event_interrupt_class_cik,
+   .num_of_watch_points = 4,
+   .mqd_size_aligned = MQD_SIZE_ALIGNED,
+   .supports_cwsr = true,
+   .needs_iommu_device = false,
+   .needs_pci_atomics = true,
+   .num_sdma_engines = 2,
+   .num_sdma_queues_per_engine = 2,
+};
+
 static const struct kfd_device_info vega10_device_info = {
.asic_family = CHIP_VEGA10,
.max_pasid_bits = 16,
@@ -237,6 +253,22 @@ static const struct kfd_device_info vega10_vf_device_info 
= {
.num_sdma_queues_per_engine = 2,
 };
 
+static const struct kfd_device_info vega12_device_info = {
+   .asic_family = CHIP_VEGA12,
+   .max_pasid_bits = 16,
+   .max_no_of_hqd  = 24,
+   .doorbell_size  = 8,
+   .ih_ring_entry_size = 8 * sizeof(uint32_t),
+   .event_interrupt_class = &event_interrupt_class_v9,
+   .num_of_watch_points = 4,

[PATCH 3/9] drm/amdkfd: Adjust the debug message in KFD ISR

[Kuehling, Felix]

From: Yong Zhao 

This makes debug message get printed even when there is early return.

Signed-off-by: Yong Zhao 
Reviewed-by: Felix Kuehling 
Signed-off-by: Felix Kuehling 
---
 drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c | 14 +++---
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c 
b/drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c
index f836897..8497864 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c
@@ -39,20 +39,20 @@ static bool event_interrupt_isr_v9(struct kfd_dev *dev,
vmid > dev->vm_info.last_vmid_kfd)
return 0;
 
-   /* If there is no valid PASID, it's likely a firmware bug */
-   pasid = SOC15_PASID_FROM_IH_ENTRY(ih_ring_entry);
-   if (WARN_ONCE(pasid == 0, "FW bug: No PASID in KFD interrupt"))
-   return 0;
-
source_id = SOC15_SOURCE_ID_FROM_IH_ENTRY(ih_ring_entry);
client_id = SOC15_CLIENT_ID_FROM_IH_ENTRY(ih_ring_entry);
+   pasid = SOC15_PASID_FROM_IH_ENTRY(ih_ring_entry);
 
-   pr_debug("client id 0x%x, source id %d, pasid 0x%x. raw data:\n",
-client_id, source_id, pasid);
+   pr_debug("client id 0x%x, source id %d, vmid %d, pasid 0x%x. raw 
data:\n",
+client_id, source_id, vmid, pasid);
pr_debug("%8X, %8X, %8X, %8X, %8X, %8X, %8X, %8X.\n",
 data[0], data[1], data[2], data[3],
 data[4], data[5], data[6], data[7]);
 
+   /* If there is no valid PASID, it's likely a firmware bug */
+   if (WARN_ONCE(pasid == 0, "FW bug: No PASID in KFD interrupt"))
+   return 0;
+
/* Interrupt types we care about: various signals and faults.
 * They will be forwarded to a work queue (see below).
 */
-- 
2.7.4

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 8/9] drm/amdgpu: Fix KFD doorbell SG BO mapping

[Kuehling, Felix]

This change prepares for adding SG BOs that will be used for mapping
doorbells into GPUVM address space.

This type of BO would be mistaken for an invalid userptr BO. Improve
that check to test that it's actually a userptr BO so that SG BOs that
are still in the CPU domain can be validated and mapped correctly.

Signed-off-by: Felix Kuehling 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
index 572ac5f..84e4c1e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
@@ -1409,7 +1409,8 @@ int amdgpu_amdkfd_gpuvm_map_memory_to_gpu(
 * the queues are still stopped and we can leave mapping for
 * the next restore worker
 */
-   if (bo->tbo.mem.mem_type == TTM_PL_SYSTEM)
+   if (amdgpu_ttm_tt_get_usermm(bo->tbo.ttm) &&
+   bo->tbo.mem.mem_type == TTM_PL_SYSTEM)
is_invalid_userptr = true;
 
if (check_if_add_bo_to_vm(avm, mem)) {
-- 
2.7.4

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 4/9] drm/amdkfd: Workaround PASID missing in gfx9 interrupt payload under non HWS

[Kuehling, Felix]

From: Yong Zhao 

This is a known gfx9 HW issue, and this change can perfectly workaround
the issue.

Signed-off-by: Yong Zhao 
Reviewed-by: Felix Kuehling 
Signed-off-by: Felix Kuehling 
---
 drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c | 25 ++---
 1 file changed, 22 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c 
b/drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c
index 8497864..a85904a 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c
@@ -23,7 +23,7 @@
 #include "kfd_priv.h"
 #include "kfd_events.h"
 #include "soc15_int.h"
-
+#include "kfd_device_queue_manager.h"
 
 static bool event_interrupt_isr_v9(struct kfd_dev *dev,
const uint32_t *ih_ring_entry,
@@ -43,14 +43,33 @@ static bool event_interrupt_isr_v9(struct kfd_dev *dev,
client_id = SOC15_CLIENT_ID_FROM_IH_ENTRY(ih_ring_entry);
pasid = SOC15_PASID_FROM_IH_ENTRY(ih_ring_entry);
 
+   /* This is a known issue for gfx9. Under non HWS, pasid is not set
+* in the interrupt payload, so we need to find out the pasid on our
+* own.
+*/
+   if (!pasid && dev->dqm->sched_policy == KFD_SCHED_POLICY_NO_HWS) {
+   const uint32_t pasid_mask = 0x;
+
+   *patched_flag = true;
+   memcpy(patched_ihre, ih_ring_entry,
+   dev->device_info->ih_ring_entry_size);
+
+   pasid = dev->kfd2kgd->get_atc_vmid_pasid_mapping_pasid(
+   dev->kgd, vmid);
+
+   /* Patch the pasid field */
+   patched_ihre[3] = cpu_to_le32((le32_to_cpu(patched_ihre[3])
+   & ~pasid_mask) | pasid);
+   }
+
pr_debug("client id 0x%x, source id %d, vmid %d, pasid 0x%x. raw 
data:\n",
 client_id, source_id, vmid, pasid);
pr_debug("%8X, %8X, %8X, %8X, %8X, %8X, %8X, %8X.\n",
 data[0], data[1], data[2], data[3],
 data[4], data[5], data[6], data[7]);
 
-   /* If there is no valid PASID, it's likely a firmware bug */
-   if (WARN_ONCE(pasid == 0, "FW bug: No PASID in KFD interrupt"))
+   /* If there is no valid PASID, it's likely a bug */
+   if (WARN_ONCE(pasid == 0, "Bug: No PASID in KFD interrupt"))
return 0;
 
/* Interrupt types we care about: various signals and faults.
-- 
2.7.4

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 9/9] drm/amdkfd: change system memory overcommit limit

[Kuehling, Felix]

From: Eric Huang 

It is to improve system limit by:
1. replacing userptrlimit with a total memory limit that
conunts TTM memory usage and userptr usage.
2. counting acc size for all BOs.

Signed-off-by: Eric Huang 
Reviewed-by: Felix Kuehling 
Signed-off-by: Felix Kuehling 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 99 ++--
 1 file changed, 58 insertions(+), 41 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
index 84e4c1e..f3129b9 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
@@ -46,9 +46,9 @@
 /* Impose limit on how much memory KFD can use */
 static struct {
uint64_t max_system_mem_limit;
-   uint64_t max_userptr_mem_limit;
+   uint64_t max_ttm_mem_limit;
int64_t system_mem_used;
-   int64_t userptr_mem_used;
+   int64_t ttm_mem_used;
spinlock_t mem_limit_lock;
 } kfd_mem_limit;
 
@@ -90,8 +90,8 @@ static bool check_if_add_bo_to_vm(struct amdgpu_vm *avm,
 }
 
 /* Set memory usage limits. Current, limits are
- *  System (kernel) memory - 3/8th System RAM
- *  Userptr memory - 3/4th System RAM
+ *  System (TTM + userptr) memory - 3/4th System RAM
+ *  TTM memory - 3/8th System RAM
  */
 void amdgpu_amdkfd_gpuvm_init_mem_limits(void)
 {
@@ -103,48 +103,54 @@ void amdgpu_amdkfd_gpuvm_init_mem_limits(void)
mem *= si.mem_unit;
 
spin_lock_init(&kfd_mem_limit.mem_limit_lock);
-   kfd_mem_limit.max_system_mem_limit = (mem >> 1) - (mem >> 3);
-   kfd_mem_limit.max_userptr_mem_limit = mem - (mem >> 2);
-   pr_debug("Kernel memory limit %lluM, userptr limit %lluM\n",
+   kfd_mem_limit.max_system_mem_limit = (mem >> 1) + (mem >> 2);
+   kfd_mem_limit.max_ttm_mem_limit = (mem >> 1) - (mem >> 3);
+   pr_debug("Kernel memory limit %lluM, TTM limit %lluM\n",
(kfd_mem_limit.max_system_mem_limit >> 20),
-   (kfd_mem_limit.max_userptr_mem_limit >> 20));
+   (kfd_mem_limit.max_ttm_mem_limit >> 20));
 }
 
 static int amdgpu_amdkfd_reserve_system_mem_limit(struct amdgpu_device *adev,
- uint64_t size, u32 domain)
+   uint64_t size, u32 domain, bool sg)
 {
-   size_t acc_size;
+   size_t acc_size, system_mem_needed, ttm_mem_needed;
int ret = 0;
 
acc_size = ttm_bo_dma_acc_size(&adev->mman.bdev, size,
   sizeof(struct amdgpu_bo));
 
spin_lock(&kfd_mem_limit.mem_limit_lock);
+
if (domain == AMDGPU_GEM_DOMAIN_GTT) {
-   if (kfd_mem_limit.system_mem_used + (acc_size + size) >
-   kfd_mem_limit.max_system_mem_limit) {
-   ret = -ENOMEM;
-   goto err_no_mem;
-   }
-   kfd_mem_limit.system_mem_used += (acc_size + size);
-   } else if (domain == AMDGPU_GEM_DOMAIN_CPU) {
-   if ((kfd_mem_limit.system_mem_used + acc_size >
-   kfd_mem_limit.max_system_mem_limit) ||
-   (kfd_mem_limit.userptr_mem_used + (size + acc_size) >
-   kfd_mem_limit.max_userptr_mem_limit)) {
-   ret = -ENOMEM;
-   goto err_no_mem;
-   }
-   kfd_mem_limit.system_mem_used += acc_size;
-   kfd_mem_limit.userptr_mem_used += size;
+   /* TTM GTT memory */
+   system_mem_needed = acc_size + size;
+   ttm_mem_needed = acc_size + size;
+   } else if (domain == AMDGPU_GEM_DOMAIN_CPU && !sg) {
+   /* Userptr */
+   system_mem_needed = acc_size + size;
+   ttm_mem_needed = acc_size;
+   } else {
+   /* VRAM and SG */
+   system_mem_needed = acc_size;
+   ttm_mem_needed = acc_size;
+   }
+
+   if ((kfd_mem_limit.system_mem_used + system_mem_needed >
+   kfd_mem_limit.max_system_mem_limit) ||
+   (kfd_mem_limit.ttm_mem_used + ttm_mem_needed >
+   kfd_mem_limit.max_ttm_mem_limit))
+   ret = -ENOMEM;
+   else {
+   kfd_mem_limit.system_mem_used += system_mem_needed;
+   kfd_mem_limit.ttm_mem_used += ttm_mem_needed;
}
-err_no_mem:
+
spin_unlock(&kfd_mem_limit.mem_limit_lock);
return ret;
 }
 
 static void unreserve_system_mem_limit(struct amdgpu_device *adev,
-  uint64_t size, u32 domain)
+   uint64_t size, u32 domain, bool sg)
 {
size_t acc_size;
 
@@ -154,14 +160,18 @@ static void unreserve_system_mem_limit(struct 
amdgpu_device *adev,
spin_lock(&kfd_mem_limit.mem_limit_lock);
if (domain == AMDGPU_GEM_DOMAIN_GTT) {
kfd_mem_limit.system_mem_used -= (acc_size + size);
-   } else if (domain == AMDGP

Re: [PATCH] drm/amdgpu: disable page queue on Vega10 SR-IOV VF

[Kuehling, Felix]

[+Philip]

On 2018-11-07 12:25 a.m., Zhang, Jerry(Junwei) wrote:
> On 11/7/18 1:15 PM, Trigger Huang wrote:
>> Currently, SDMA page queue is not used under SR-IOV VF, and this
>> queue will
>> cause ring test failure in amdgpu module reload case. So just disable
>> it.
>>
>> Signed-off-by: Trigger Huang 
>
> Looks we ran into several issues about it on vega.
> kfd also disabled vega10 for development.(but not sure the detail
> issue for them)
>
> Thus, we may disable it for vega10 as well?
> any comment? Alex, Christian, Flex.

We ran into a regression with the page queue in a specific KFDTest that
runs user mode SDMA in two processes. The SDMA engine would stall for
about 6 seconds after one of the processes terminates (and destroys its
queues). We don't have a root cause. Suspect an SDMA firmware issue.

Regards,
  Felix


>
> Regards,
> Jerry
>> ---
>>   drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c | 4 +++-
>>   1 file changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c
>> b/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c
>> index e39a09eb0f..4edc848 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c
>> @@ -1451,7 +1451,9 @@ static int sdma_v4_0_early_init(void *handle)
>>   adev->sdma.has_page_queue = false;
>>   } else {
>>   adev->sdma.num_instances = 2;
>> -    if (adev->asic_type != CHIP_VEGA20 &&
>> +    if ((adev->asic_type == CHIP_VEGA10) &&
>> amdgpu_sriov_vf((adev)))
>> +    adev->sdma.has_page_queue = false;
>> +    else if (adev->asic_type != CHIP_VEGA20 &&
>>   adev->asic_type != CHIP_VEGA12)
>>   adev->sdma.has_page_queue = true;
>>   }
>
> ___
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdgpu: fix huge page handling on Vega10

[Kuehling, Felix]

On 2018-11-12 12:09 p.m., Christian König wrote:
> We accidentially set the huge flag on the parent instead of the childs.
> This caused some VM faults under memory pressure.

Reviewed-by: Felix Kuehling 

I got a bit confused when re-reading this code. Maybe part of it is that
cursor.entry is not really an entry (as in PTE), but a page table.
cursor.pfn identifies the entry within the table. Maybe renaming
cursor.entry to something like cursor.table would make it slightly more
intuitive.

Regards,
  Felix


>
> Signed-off-by: Christian König 
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 18 ++
>  1 file changed, 10 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index 352b30409060..dad0e2342df9 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -1632,13 +1632,6 @@ static int amdgpu_vm_update_ptes(struct 
> amdgpu_pte_update_params *params,
>   continue;
>   }
>  
> - /* First check if the entry is already handled */
> - if (cursor.pfn < frag_start) {
> - cursor.entry->huge = true;
> - amdgpu_vm_pt_next(adev, &cursor);
> - continue;
> - }
> -
>   /* If it isn't already handled it can't be a huge page */
>   if (cursor.entry->huge) {
>   /* Add the entry to the relocated list to update it. */
> @@ -1701,8 +1694,17 @@ static int amdgpu_vm_update_ptes(struct 
> amdgpu_pte_update_params *params,
>   }
>   } while (frag_start < entry_end);
>  
> - if (frag >= shift)
> + if (amdgpu_vm_pt_descendant(adev, &cursor)) {
> + /* Mark all child entries as huge */
> + while (cursor.pfn < frag_start) {
> + cursor.entry->huge = true;
> + amdgpu_vm_pt_next(adev, &cursor);
> + }
> +
> + } else if (frag >= shift) {
> + /* or just move on to the next on the same level. */
>   amdgpu_vm_pt_next(adev, &cursor);
> + }
>   }
>  
>   return 0;
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH] drm/amdgpu : Use XGMI mapping when devices on the same hive v2

[Kuehling, Felix]

This change is not suitable for amd-staging-drm-next. PCIe P2P was not enabled 
on amd-staging-drm-next because it's not reliable yet. This change enables it 
even in situations that are not safe (including small BAR systems).

Why are you porting this change to amd-staging-drm-next? Does anyone depend on 
XGMI support on this branch?

Regards,
  Felix

-Original Message-
From: amd-gfx  On Behalf Of Russell, Kent
Sent: Thursday, November 15, 2018 11:54 AM
To: amd-gfx@lists.freedesktop.org
Cc: Russell, Kent ; Liu, Shaoyun 
Subject: [PATCH] drm/amdgpu : Use XGMI mapping when devices on the same hive v2

From: Shaoyun Liu 

VM mapping will only fall back to P2P if XGMI mapping is not available

V2: Rebase onto 4.20

Change-Id: I7a854ab3d5c9958bd45d4fe439ea7e370a092e7a
Signed-off-by: Shaoyun Liu 
Reviewed-by: Felix Kuehling 
Reviewed-by: Huang Rui 
Reviewed-by: Christian König 
Signed-off-by: Kent Russell 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 16 ++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index dad0e23..576d168 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -2011,6 +2011,8 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev,
struct drm_mm_node *nodes;
struct dma_fence *exclusive, **last_update;
uint64_t flags;
+   uint64_t vram_base_offset = adev->vm_manager.vram_base_offset;
+   struct amdgpu_device *bo_adev;
int r;
 
if (clear || !bo) {
@@ -2029,9 +2031,19 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev,
exclusive = reservation_object_get_excl(bo->tbo.resv);
}
 
-   if (bo)
+   if (bo) {
flags = amdgpu_ttm_tt_pte_flags(adev, bo->tbo.ttm, mem);
-   else
+   bo_adev = amdgpu_ttm_adev(bo->tbo.bdev);
+   if (mem && mem->mem_type == TTM_PL_VRAM && adev != bo_adev) {
+   if (adev->gmc.xgmi.hive_id &&
+   adev->gmc.xgmi.hive_id == 
bo_adev->gmc.xgmi.hive_id) {
+   vram_base_offset = 
bo_adev->vm_manager.vram_base_offset;
+   } else {
+   flags |= AMDGPU_PTE_SYSTEM;
+   vram_base_offset = bo_adev->gmc.aper_base;
+   }
+   }
+   } else
flags = 0x0;
 
if (clear || (bo && bo->tbo.resv == vm->root.base.bo->tbo.resv))
-- 
2.7.4

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH] drm/amdgpu: enable paging queue doorbell support

[Kuehling, Felix]

You changed the doorbell routing in NBIO. I think that won't work for SR-IOV, 
because it's not controlled by the guest OS there. We may need to disable 
paging queue doorbell on Vega10 and Vega12 with SRIOV. For Vega20 we plan to 
change the doorbell layout before it goes to production (Oak started working on 
that). So on Vega20 we should be able to enable the doorbell for the paging 
queue.

Regards,
  Felix

-Original Message-
From: amd-gfx  On Behalf Of Yang, Philip
Sent: Thursday, November 15, 2018 12:54 PM
To: Alex Deucher 
Cc: amd-gfx list 
Subject: Re: [PATCH] drm/amdgpu: enable paging queue doorbell support

On 2018-11-15 11:43 a.m., Alex Deucher wrote:
> On Thu, Nov 15, 2018 at 11:08 AM Yang, Philip  wrote:
>> paging queues doorbell index use existing assignment 
>> sDMA_HI_PRI_ENGINE0/1 index, and increase SDMA_DOORBELL_RANGE size 
>> from 2 dwords to 4 dwords to enable the new doorbell index.
>>
>> Change-Id: I9adb965f16ee4089d261d9a22231337739184e49
>> Signed-off-by: Philip Yang 
> Is there a specific fw version requirement for this?  If so, we need 
> to add a check.  Also will this break SR-IOV due to the doorbell 
> mapping requirements for other OSes?  Have we resolved that yet?
>
Yes, new FW is required to enable paging queue, it's good idea to check FW 
version when we enable paging queue later after new FW is checked in. paging 
queue is not enabled yet.
This change will not take effect until paging queue is enabled.

This does not change doorbell mapping so it will not break SR-IOV. My 
understanding is Doorbell mapping is good for vega10/12/20, but need update for 
future chip.
>> ---
>>   drivers/gpu/drm/amd/amdgpu/nbio_v6_1.c |  2 +-
>>   drivers/gpu/drm/amd/amdgpu/nbio_v7_4.c |  2 +-
>>   drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c | 25 +
>>   3 files changed, 19 insertions(+), 10 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/nbio_v6_1.c 
>> b/drivers/gpu/drm/amd/amdgpu/nbio_v6_1.c
>> index 6f9c54978cc1..0eb42c29ecac 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/nbio_v6_1.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/nbio_v6_1.c
>> @@ -80,7 +80,7 @@ static void nbio_v6_1_sdma_doorbell_range(struct 
>> amdgpu_device *adev, int instan
>>
>>  if (use_doorbell) {
>>  doorbell_range = REG_SET_FIELD(doorbell_range, 
>> BIF_SDMA0_DOORBELL_RANGE, OFFSET, doorbell_index);
>> -   doorbell_range = REG_SET_FIELD(doorbell_range, 
>> BIF_SDMA0_DOORBELL_RANGE, SIZE, 2);
>> +   doorbell_range = REG_SET_FIELD(doorbell_range, 
>> + BIF_SDMA0_DOORBELL_RANGE, SIZE, 4);
>>  } else
>>  doorbell_range = REG_SET_FIELD(doorbell_range, 
>> BIF_SDMA0_DOORBELL_RANGE, SIZE, 0);
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/nbio_v7_4.c 
>> b/drivers/gpu/drm/amd/amdgpu/nbio_v7_4.c
>> index f8cee95d61cc..9342ee03d7d4 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/nbio_v7_4.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/nbio_v7_4.c
>> @@ -76,7 +76,7 @@ static void nbio_v7_4_sdma_doorbell_range(struct 
>> amdgpu_device *adev, int instan
>>
>>  if (use_doorbell) {
>>  doorbell_range = REG_SET_FIELD(doorbell_range, 
>> BIF_SDMA0_DOORBELL_RANGE, OFFSET, doorbell_index);
>> -   doorbell_range = REG_SET_FIELD(doorbell_range, 
>> BIF_SDMA0_DOORBELL_RANGE, SIZE, 2);
>> +   doorbell_range = REG_SET_FIELD(doorbell_range, 
>> + BIF_SDMA0_DOORBELL_RANGE, SIZE, 4);
>>  } else
>>  doorbell_range = REG_SET_FIELD(doorbell_range, 
>> BIF_SDMA0_DOORBELL_RANGE, SIZE, 0);
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c 
>> b/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c
>> index f4490cdd9804..96c9e83204b7 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c
>> @@ -925,11 +925,9 @@ static void sdma_v4_0_page_resume(struct amdgpu_device 
>> *adev, unsigned int i)
>>  OFFSET, ring->doorbell_index);
>>  WREG32_SDMA(i, mmSDMA0_PAGE_DOORBELL, doorbell);
>>  WREG32_SDMA(i, mmSDMA0_PAGE_DOORBELL_OFFSET, doorbell_offset);
>> -   /* TODO: enable doorbell support */
>> -   /*adev->nbio_funcs->sdma_doorbell_range(adev, i, ring->use_doorbell,
>> - ring->doorbell_index);*/
>>
>> -   sdma_v4_0_ring_set_wptr(ring);
>> +   /* paging queue doorbell index is already enabled at 
>> sdma_v4_0_gfx_resume */
>> +   sdma_v4_0_page_ring_set_wptr(ring);
>>
>>  /* set minor_ptr_update to 0 after wptr programed */
>>  WREG32_SDMA(i, mmSDMA0_PAGE_MINOR_PTR_UPDATE, 0); @@ -1504,9 
>> +1502,6 @@ static int sdma_v4_0_sw_init(void *handle)
>>  ring->ring_obj = NULL;
>>  ring->use_doorbell = true;
>>
>> -   DRM_INFO("use_doorbell being set to: [%s]\n",
>> -   ring->use_doorbell?"true":"false");
>> -
>>  if (adev->asic_type == CHIP_VEGA10

RE: [PATCH] drm/amdgpu : Use XGMI mapping when devices on the same hive v2

[Kuehling, Felix]

Hmm, we should remove the PCIe portion of this change. We just added some extra 
checks there on amd-kfd-staging that should make it closer to upstreamable. For 
now, just handle the XGMI case, but return -EINVAL in the else-branch (for 
other remote VRAM cases).

Regards,
  Felix

From: Russell, Kent
Sent: Thursday, November 15, 2018 1:04 PM
To: Kuehling, Felix ; amd-gfx@lists.freedesktop.org
Cc: Liu, Shaoyun 
Subject: Re: [PATCH] drm/amdgpu : Use XGMI mapping when devices on the same 
hive v2

It was merged to 4.19 on Sept 21. It got missed on the 4.20 rebase.

Kent

KENT RUSSELL
Sr. Software Engineer | Linux Compute Kernel
1 Commerce Valley Drive East
Markham, ON L3T 7X6
O +(1) 289-695-2122 | Ext 72122

From: Kuehling, Felix
Sent: Thursday, November 15, 2018 12:57:44 PM
To: Russell, Kent; 
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
Cc: Russell, Kent; Liu, Shaoyun
Subject: RE: [PATCH] drm/amdgpu : Use XGMI mapping when devices on the same 
hive v2

This change is not suitable for amd-staging-drm-next. PCIe P2P was not enabled 
on amd-staging-drm-next because it's not reliable yet. This change enables it 
even in situations that are not safe (including small BAR systems).

Why are you porting this change to amd-staging-drm-next? Does anyone depend on 
XGMI support on this branch?

Regards,
  Felix

-Original Message-
From: amd-gfx 
mailto:amd-gfx-boun...@lists.freedesktop.org>>
 On Behalf Of Russell, Kent
Sent: Thursday, November 15, 2018 11:54 AM
To: amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
Cc: Russell, Kent mailto:kent.russ...@amd.com>>; Liu, 
Shaoyun mailto:shaoyun@amd.com>>
Subject: [PATCH] drm/amdgpu : Use XGMI mapping when devices on the same hive v2

From: Shaoyun Liu mailto:shaoyun@amd.com>>

VM mapping will only fall back to P2P if XGMI mapping is not available

V2: Rebase onto 4.20

Change-Id: I7a854ab3d5c9958bd45d4fe439ea7e370a092e7a
Signed-off-by: Shaoyun Liu mailto:shaoyun@amd.com>>
Reviewed-by: Felix Kuehling 
mailto:felix.kuehl...@amd.com>>
Reviewed-by: Huang Rui mailto:ray.hu...@amd.com>>
Reviewed-by: Christian König 
mailto:christian.koe...@amd.com>>
Signed-off-by: Kent Russell mailto:kent.russ...@amd.com>>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 16 ++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index dad0e23..576d168 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -2011,6 +2011,8 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev,
 struct drm_mm_node *nodes;
 struct dma_fence *exclusive, **last_update;
 uint64_t flags;
+   uint64_t vram_base_offset = adev->vm_manager.vram_base_offset;
+   struct amdgpu_device *bo_adev;
 int r;

 if (clear || !bo) {
@@ -2029,9 +2031,19 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev,
 exclusive = reservation_object_get_excl(bo->tbo.resv);
 }

-   if (bo)
+   if (bo) {
 flags = amdgpu_ttm_tt_pte_flags(adev, bo->tbo.ttm, mem);
-   else
+   bo_adev = amdgpu_ttm_adev(bo->tbo.bdev);
+   if (mem && mem->mem_type == TTM_PL_VRAM && adev != bo_adev) {
+   if (adev->gmc.xgmi.hive_id &&
+   adev->gmc.xgmi.hive_id == 
bo_adev->gmc.xgmi.hive_id) {
+   vram_base_offset = 
bo_adev->vm_manager.vram_base_offset;
+   } else {
+   flags |= AMDGPU_PTE_SYSTEM;
+   vram_base_offset = bo_adev->gmc.aper_base;
+   }
+   }
+   } else
 flags = 0x0;

 if (clear || (bo && bo->tbo.resv == vm->root.base.bo->tbo.resv))
--
2.7.4

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH] drm/amdgpu : Use XGMI mapping when devices on the same hive v3

[Kuehling, Felix]

Sorry, something is still missing here. The new variable vram_base_offset isn't 
used anywhere. We have some other changes in amd-kfd-staging to use that 
vram_base_offset that are probably missing on amd-staging-drm-next. This change 
won't have any effect as is.

Regards,
  Felix

-Original Message-
From: amd-gfx  On Behalf Of Russell, Kent
Sent: Thursday, November 15, 2018 2:23 PM
To: amd-gfx@lists.freedesktop.org
Cc: Russell, Kent ; Liu, Shaoyun 
Subject: [PATCH] drm/amdgpu : Use XGMI mapping when devices on the same hive v3

From: Shaoyun Liu 

VM mapping will only fall back to P2P if XGMI mapping is not available

V2: Rebase onto 4.20
V3: Remove PCIe-related things

Change-Id: I7a854ab3d5c9958bd45d4fe439ea7e370a092e7a
Signed-off-by: Shaoyun Liu 
Reviewed-by: Felix Kuehling 
Reviewed-by: Huang Rui 
Reviewed-by: Christian König 
Signed-off-by: Kent Russell 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 12 ++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index dad0e23..5aa60bf 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -2011,6 +2011,8 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev,
struct drm_mm_node *nodes;
struct dma_fence *exclusive, **last_update;
uint64_t flags;
+   uint64_t vram_base_offset = adev->vm_manager.vram_base_offset;
+   struct amdgpu_device *bo_adev;
int r;
 
if (clear || !bo) {
@@ -2029,9 +2031,15 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev,
exclusive = reservation_object_get_excl(bo->tbo.resv);
}
 
-   if (bo)
+   if (bo) {
flags = amdgpu_ttm_tt_pte_flags(adev, bo->tbo.ttm, mem);
-   else
+   bo_adev = amdgpu_ttm_adev(bo->tbo.bdev);
+   if (adev->gmc.xgmi.hive_id &&
+   adev->gmc.xgmi.hive_id == bo_adev->gmc.xgmi.hive_id)
+   vram_base_offset = bo_adev->vm_manager.vram_base_offset;
+   else
+   return -EINVAL;
+   } else
flags = 0x0;
 
if (clear || (bo && bo->tbo.resv == vm->root.base.bo->tbo.resv))
-- 
2.7.4

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH] drm/amdgpu: Fix Kernel Oops triggered by kfdtest

[Kuehling, Felix]

Apologies. We already have a fix for this on our internal amd-kfd-staging 
branch, but it's missing from amd-staging-drm-next. I'll cherry-pick our fix to 
amd-staging-drm-next and nominate it for drm-fixes.

Regards,
  Felix

-Original Message-
From: amd-gfx  On Behalf Of Joerg Roedel
Sent: Thursday, November 15, 2018 9:46 AM
To: Zhu, Rex ; Quan, Evan ; Deucher, 
Alexander ; Koenig, Christian 
; Zhou, David(ChunMing) 
Cc: Joerg Roedel ; dri-de...@lists.freedesktop.org; 
amd-gfx@lists.freedesktop.org; linux-ker...@vger.kernel.org
Subject: [PATCH] drm/amdgpu: Fix Kernel Oops triggered by kfdtest

From: Joerg Roedel 

Running kfdtest on Kaveri triggers a kernel NULL-ptr dereference:

BUG: unable to handle kernel NULL pointer dereference at 

PGD 42c017067 P4D 42c017067 PUD 40f071067 PMD 0
Oops: 0010 [#1] SMP NOPTI
CPU: 0 PID: 13107 Comm: kfdtest Not tainted 4.20.0-rc2+ #11
Hardware name: Gigabyte Technology Co., Ltd. To be filled by 
O.E.M./F2A88XM-HD3, BIOS F6 05/28/2014
RIP: 0010:  (null)
Code: Bad RIP value.
RSP: 0018:c90001adbbf0 EFLAGS: 00010202
RAX: a0806240 RBX: 88842a0fbc00 RCX: 0002
RDX: 0001 RSI: 0004 RDI: 88842969
RBP: c90001adbbf8 R08: 2000 R09: 88842e542ec0
R10: 7feff778f008 R11: 7feff778f010 R12: 
R13: 88840f063a20 R14: 88842a0fbd20 R15: 0f43ff60
FS:  7feff7769740() GS:88842fa0() 
knlGS:
CS:  0010 DS:  ES:  CR0: 80050033
CR2: ffd6 CR3: 00040f122000 CR4: 000406f0
Call Trace:
 ? amdgpu_amdkfd_set_compute_idle+0x29/0x30 [amdgpu]
 register_process+0x140/0x150 [amdgpu]
 pqm_create_queue+0x395/0x560 [amdgpu]
 kfd_ioctl_create_queue+0x285/0x680 [amdgpu]
 kfd_ioctl+0x27f/0x450 [amdgpu]
 ? kfd_ioctl_destroy_queue+0x80/0x80 [amdgpu]
 do_vfs_ioctl+0x96/0x6a0
 ? __audit_syscall_entry+0xdd/0x130
 ? handle_mm_fault+0x11b/0x240
 ksys_ioctl+0x67/0x90
 __x64_sys_ioctl+0x1a/0x20
 do_syscall_64+0x61/0x190
 entry_SYSCALL_64_after_hwframe+0x44/0xa9

The reason is that the pp_funcs->switch_power_profile function pointer is not 
set for a Kaveri ASIC and thus the kernel calls a NULL pointer.

Add a check before calling the function to avoid that.

Signed-off-by: Joerg Roedel 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_dpm.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_dpm.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_dpm.h
index f972cd156795..0ecedd30f2aa 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_dpm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_dpm.h
@@ -337,8 +337,9 @@ enum amdgpu_pcie_gen {
(adev)->powerplay.pp_handle, request))
 
 #define amdgpu_dpm_switch_power_profile(adev, type, en) \
-   ((adev)->powerplay.pp_funcs->switch_power_profile(\
-   (adev)->powerplay.pp_handle, type, en))
+   if ((adev)->powerplay.pp_funcs->switch_power_profile != NULL) \
+   ((adev)->powerplay.pp_funcs->switch_power_profile(\
+   (adev)->powerplay.pp_handle, type, en))
 
 #define amdgpu_dpm_set_clockgating_by_smu(adev, msg_id) \
((adev)->powerplay.pp_funcs->set_clockgating_by_smu(\
--
2.13.7

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH] drm/amdgpu: Fix oops when pp_funcs->switch_power_profile is unset

[Kuehling, Felix]

On Vega20 and other pre-production GPUs, powerplay is not enabled yet.
Check for NULL pointers before calling pp_funcs function pointers.

Also affects Kaveri.

CC: Joerg Roedel 
Signed-off-by: Felix Kuehling 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
index a9c7597..1c1fed6 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
@@ -503,8 +503,11 @@ void amdgpu_amdkfd_set_compute_idle(struct kgd_dev *kgd, 
bool idle)
 {
struct amdgpu_device *adev = (struct amdgpu_device *)kgd;
 
-   amdgpu_dpm_switch_power_profile(adev,
-   PP_SMC_POWER_PROFILE_COMPUTE, !idle);
+   if (adev->powerplay.pp_funcs &&
+   adev->powerplay.pp_funcs->switch_power_profile)
+   amdgpu_dpm_switch_power_profile(adev,
+   PP_SMC_POWER_PROFILE_COMPUTE,
+   !idle);
 }
 
 bool amdgpu_amdkfd_is_kfd_vmid(struct amdgpu_device *adev, u32 vmid)
-- 
2.7.4

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdgpu: enable paging queue doorbell support v3

[Kuehling, Felix]

Looks good to me. Reviewed-by: Felix Kuehling 

I hope Alex or Christian can also review this in case I'm missing
something about how doorbells are used in amdgpu.

Regards,
  Felix

On 2018-11-16 2:08 p.m., Yang, Philip wrote:
> Because increase SDMA_DOORBELL_RANGE to add new SDMA doorbell for paging 
> queue will
> break SRIOV, instead we can reserve and map two doorbell pages for amdgpu, 
> paging
> queues doorbell index use same index as SDMA gfx queues index but on second 
> page.
>
> For Vega20, after we change doorbell layout to increase SDMA doorbell for 8 
> SDMA RLC
> queues later, we could use new doorbell index for paging queue.
>
> Change-Id: I9adb965f16ee4089d261d9a22231337739184e49
> Signed-off-by: Philip Yang 
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 13 +
>  drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c | 33 +-
>  2 files changed, 33 insertions(+), 13 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 590588a82471..7a54760591ae 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -534,6 +534,19 @@ static int amdgpu_device_doorbell_init(struct 
> amdgpu_device *adev)
>   if (adev->doorbell.num_doorbells == 0)
>   return -EINVAL;
>  
> + /* For Vega, reserve and map two pages on doorbell BAR since SDMA
> +  * paging queue doorbell use the second page
> +  */
> + switch (adev->asic_type) {
> + case CHIP_VEGA10:
> + case CHIP_VEGA12:
> + case CHIP_VEGA20:
> + adev->doorbell.num_doorbells *= 2;
> + break;
> + default:
> + break;
> + }
> +
>   adev->doorbell.ptr = ioremap(adev->doorbell.base,
>adev->doorbell.num_doorbells *
>sizeof(u32));
> diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c 
> b/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c
> index f4490cdd9804..db5a71db9f10 100644
> --- a/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c
> @@ -925,11 +925,9 @@ static void sdma_v4_0_page_resume(struct amdgpu_device 
> *adev, unsigned int i)
>   OFFSET, ring->doorbell_index);
>   WREG32_SDMA(i, mmSDMA0_PAGE_DOORBELL, doorbell);
>   WREG32_SDMA(i, mmSDMA0_PAGE_DOORBELL_OFFSET, doorbell_offset);
> - /* TODO: enable doorbell support */
> - /*adev->nbio_funcs->sdma_doorbell_range(adev, i, ring->use_doorbell,
> -   ring->doorbell_index);*/
>  
> - sdma_v4_0_ring_set_wptr(ring);
> + /* paging queue doorbell index is already enabled at 
> sdma_v4_0_gfx_resume */
> + sdma_v4_0_page_ring_set_wptr(ring);
>  
>   /* set minor_ptr_update to 0 after wptr programed */
>   WREG32_SDMA(i, mmSDMA0_PAGE_MINOR_PTR_UPDATE, 0);
> @@ -1504,18 +1502,14 @@ static int sdma_v4_0_sw_init(void *handle)
>   ring->ring_obj = NULL;
>   ring->use_doorbell = true;
>  
> - DRM_INFO("use_doorbell being set to: [%s]\n",
> - ring->use_doorbell?"true":"false");
> -
>   if (adev->asic_type == CHIP_VEGA10)
>   ring->doorbell_index = (i == 0) ?
> - (AMDGPU_VEGA10_DOORBELL64_sDMA_ENGINE0 << 1) 
> //get DWORD offset
> - : (AMDGPU_VEGA10_DOORBELL64_sDMA_ENGINE1 << 1); 
> // get DWORD offset
> + (AMDGPU_VEGA10_DOORBELL64_sDMA_ENGINE0 << 1) // 
> doorbell size 2 dwords
> + : (AMDGPU_VEGA10_DOORBELL64_sDMA_ENGINE1 << 1); 
> // doorbell size 2 dwords
>   else
>   ring->doorbell_index = (i == 0) ?
> - (AMDGPU_DOORBELL64_sDMA_ENGINE0 << 1) //get 
> DWORD offset
> - : (AMDGPU_DOORBELL64_sDMA_ENGINE1 << 1); // get 
> DWORD offset
> -
> + (AMDGPU_DOORBELL64_sDMA_ENGINE0 << 1) // 
> doorbell size 2 dwords
> + : (AMDGPU_DOORBELL64_sDMA_ENGINE1 << 1); // 
> doorbell size 2 dwords
>  
>   sprintf(ring->name, "sdma%d", i);
>   r = amdgpu_ring_init(adev, ring, 1024,
> @@ -1529,7 +1523,20 @@ static int sdma_v4_0_sw_init(void *handle)
>   if (adev->sdma.has_page_queue) {
>   ring = &adev->sdma.instance[i].page;
>   ring->ring_obj = NULL;
> - ring->use_doorbell = false;
> + ring->use_doorbell = true;
> +
> + /* paging queue use same doorbell index/routing as gfx 
> queue
> +  * with 0x400 (4096 dwords) offset on second doorbell 
> page
> +  */
> + if (adev->asic_type == CHIP_VEGA10)
> + ring->doorbell_index = (i == 0) ?

Re: [PATCH 7/9] drm/amdkfd: Fix and simplify sync object handling for KFD

[Kuehling, Felix]

Hi Christian,

Would you review this patch? Just looking at the code, calling
amdgpu_sync_fence with adev=NULL should be OK for us. It's just a bit
unusual compared to amdgpu's usage of this function. We've had this
patch in kfd-staging for a while without problems. If you're OK with
this I'll go ahead and push this upstream as well.

Thanks,
  Felix

On 2018-11-05 8:40 p.m., Kuehling, Felix wrote:
> The adev parameter in amdgpu_sync_fence and amdgpu_sync_resv is only
> needed for updating sync->last_vm_update. This breaks if different
> adevs are passed to calls for the same sync object.
>
> Always pass NULL for calls from KFD because sync objects used for
> KFD don't belong to any particular device, and KFD doesn't need the
> sync->last_vm_update fence.
>
> This fixes kernel log warnings on multi-GPU systems after recent
> changes in amdgpu_amdkfd_gpuvm_restore_process_bos.
>
> Signed-off-by: Felix Kuehling 
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 28 
> +---
>  1 file changed, 5 insertions(+), 23 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> index d005371..572ac5f 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> @@ -395,23 +395,6 @@ static int vm_validate_pt_pd_bos(struct amdgpu_vm *vm)
>   return 0;
>  }
>  
> -static int sync_vm_fence(struct amdgpu_device *adev, struct amdgpu_sync 
> *sync,
> -  struct dma_fence *f)
> -{
> - int ret = amdgpu_sync_fence(adev, sync, f, false);
> -
> - /* Sync objects can't handle multiple GPUs (contexts) updating
> -  * sync->last_vm_update. Fortunately we don't need it for
> -  * KFD's purposes, so we can just drop that fence.
> -  */
> - if (sync->last_vm_update) {
> - dma_fence_put(sync->last_vm_update);
> - sync->last_vm_update = NULL;
> - }
> -
> - return ret;
> -}
> -
>  static int vm_update_pds(struct amdgpu_vm *vm, struct amdgpu_sync *sync)
>  {
>   struct amdgpu_bo *pd = vm->root.base.bo;
> @@ -422,7 +405,7 @@ static int vm_update_pds(struct amdgpu_vm *vm, struct 
> amdgpu_sync *sync)
>   if (ret)
>   return ret;
>  
> - return sync_vm_fence(adev, sync, vm->last_update);
> + return amdgpu_sync_fence(NULL, sync, vm->last_update, false);
>  }
>  
>  /* add_bo_to_vm - Add a BO to a VM
> @@ -826,7 +809,7 @@ static int unmap_bo_from_gpuvm(struct amdgpu_device *adev,
>   /* Add the eviction fence back */
>   amdgpu_bo_fence(pd, &vm->process_info->eviction_fence->base, true);
>  
> - sync_vm_fence(adev, sync, bo_va->last_pt_update);
> + amdgpu_sync_fence(NULL, sync, bo_va->last_pt_update, false);
>  
>   return 0;
>  }
> @@ -851,7 +834,7 @@ static int update_gpuvm_pte(struct amdgpu_device *adev,
>   return ret;
>   }
>  
> - return sync_vm_fence(adev, sync, bo_va->last_pt_update);
> + return amdgpu_sync_fence(NULL, sync, bo_va->last_pt_update, false);
>  }
>  
>  static int map_bo_to_gpuvm(struct amdgpu_device *adev,
> @@ -911,7 +894,7 @@ static int process_sync_pds_resv(struct 
> amdkfd_process_info *process_info,
>   vm_list_node) {
>   struct amdgpu_bo *pd = peer_vm->root.base.bo;
>  
> - ret = amdgpu_sync_resv(amdgpu_ttm_adev(pd->tbo.bdev),
> + ret = amdgpu_sync_resv(NULL,
>   sync, pd->tbo.resv,
>   AMDGPU_FENCE_OWNER_UNDEFINED, false);
>   if (ret)
> @@ -2084,8 +2067,7 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, 
> struct dma_fence **ef)
>   pr_debug("Memory eviction: Validate BOs failed. Try 
> again\n");
>   goto validate_map_fail;
>   }
> - ret = amdgpu_sync_fence(amdgpu_ttm_adev(bo->tbo.bdev),
> - &sync_obj, bo->tbo.moving, false);
> + ret = amdgpu_sync_fence(NULL, &sync_obj, bo->tbo.moving, false);
>   if (ret) {
>   pr_debug("Memory eviction: Sync BO fence failed. Try 
> again\n");
>   goto validate_map_fail;
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdgpu/gfx: use proper offset define for MEC doorbells

[Kuehling, Felix]

On 2018-11-16 3:30 p.m., Alex Deucher wrote:
> Looks like a copy paste typo.
>
> Signed-off-by: Alex Deucher 

Reviewed-by: Felix Kuehling 


> ---
>  drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c 
> b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
> index c27caa144c57..e409fbf67fe7 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
> @@ -1566,7 +1566,7 @@ static int gfx_v9_0_compute_ring_init(struct 
> amdgpu_device *adev, int ring_id,
>  
>   ring->ring_obj = NULL;
>   ring->use_doorbell = true;
> - ring->doorbell_index = (AMDGPU_DOORBELL_MEC_RING0 + ring_id) << 1;
> + ring->doorbell_index = (AMDGPU_DOORBELL64_MEC_RING0 + ring_id) << 1;
>   ring->eop_gpu_addr = adev->gfx.mec.hpd_eop_gpu_addr
>   + (ring_id * GFX9_MEC_HPD_SIZE);
>   sprintf(ring->name, "comp_%d.%d.%d", ring->me, ring->pipe, ring->queue);
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdgpu: enable paging queue doorbell support v3

[Kuehling, Felix]

On 2018-11-16 4:35 p.m., Alex Deucher wrote:
>> +   ring->doorbell_index += 0x400;
> I don't quite understand how this works.  Why don't we have to adjust
> the doorbell range registers in the nbio code?

NBIO only looks at the lower 12 bits of the doorbell address. So adding
0x400 dwords or 0x1000 bytes has no effect on the doorbell routing.

That's also how we can use SDMA doorbells in our per-process doorbell
pages at higher addresses.

Regards,
  Felix


___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdgpu: Fix oops when pp_funcs->switch_power_profile is unset

[Kuehling, Felix]

I pushed the change to amd-staging-drm-next. I'd also like to get it
into drm-fixes. Should I cherry-pick it to your drm-fixes-4.20 branch on
fd.o?


Thanks,
  Felix


On 2018-11-15 4:59 p.m., Deucher, Alexander wrote:
>
> Reviewed-by: Alex Deucher 
>
> 
> *From:* amd-gfx  on behalf of
> Kuehling, Felix 
> *Sent:* Thursday, November 15, 2018 4:56:51 PM
> *To:* amd-gfx@lists.freedesktop.org
> *Cc:* Kuehling, Felix; Joerg Roedel
> *Subject:* [PATCH] drm/amdgpu: Fix oops when
> pp_funcs->switch_power_profile is unset
>  
> On Vega20 and other pre-production GPUs, powerplay is not enabled yet.
> Check for NULL pointers before calling pp_funcs function pointers.
>
> Also affects Kaveri.
>
> CC: Joerg Roedel 
> Signed-off-by: Felix Kuehling 
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 7 +--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
> index a9c7597..1c1fed6 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
> @@ -503,8 +503,11 @@ void amdgpu_amdkfd_set_compute_idle(struct
> kgd_dev *kgd, bool idle)
>  {
>  struct amdgpu_device *adev = (struct amdgpu_device *)kgd;
>  
> -   amdgpu_dpm_switch_power_profile(adev,
> -   PP_SMC_POWER_PROFILE_COMPUTE,
> !idle);
> +   if (adev->powerplay.pp_funcs &&
> +   adev->powerplay.pp_funcs->switch_power_profile)
> +   amdgpu_dpm_switch_power_profile(adev,
> +  
> PP_SMC_POWER_PROFILE_COMPUTE,
> +   !idle);
>  }
>  
>  bool amdgpu_amdkfd_is_kfd_vmid(struct amdgpu_device *adev, u32 vmid)
> -- 
> 2.7.4
>
> ___
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdgpu: enable paging queue doorbell support v2

[Kuehling, Felix]

Hi Christian,

On 2018-11-19 6:24 a.m., Christian König wrote:
> Am 15.11.18 um 20:10 schrieb Yang, Philip:
>> paging queues doorbell index use existing assignment
>> sDMA_HI_PRI_ENGINE0/1
>> index, and increase SDMA_DOORBELL_RANGE size from 2 dwords to 4
>> dwords to
>> enable the new doorbell index.
>>
>> v2: disable paging queue doorbell on Vega10 and Vega12 with SRIOV
>>
>> Change-Id: I9adb965f16ee4089d261d9a22231337739184e49
>> Signed-off-by: Philip Yang 
>
> Acked-by: Christian König 

This change was superseded by "drm/amdgpu: enable paging queue doorbell
support v3", which doesn't need to change the doorbell routing and no
special rules for SRIOV. Instead it uses two pages from the doorbell
aperture. That's OK, because KFD only uses page 2 and up.

Regards,
  Felix


>
>> ---
>>   drivers/gpu/drm/amd/amdgpu/nbio_v6_1.c |  2 +-
>>   drivers/gpu/drm/amd/amdgpu/nbio_v7_4.c |  2 +-
>>   drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c | 26 ++
>>   3 files changed, 20 insertions(+), 10 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/nbio_v6_1.c
>> b/drivers/gpu/drm/amd/amdgpu/nbio_v6_1.c
>> index 6f9c54978cc1..0eb42c29ecac 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/nbio_v6_1.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/nbio_v6_1.c
>> @@ -80,7 +80,7 @@ static void nbio_v6_1_sdma_doorbell_range(struct
>> amdgpu_device *adev, int instan
>>     if (use_doorbell) {
>>   doorbell_range = REG_SET_FIELD(doorbell_range,
>> BIF_SDMA0_DOORBELL_RANGE, OFFSET, doorbell_index);
>> -    doorbell_range = REG_SET_FIELD(doorbell_range,
>> BIF_SDMA0_DOORBELL_RANGE, SIZE, 2);
>> +    doorbell_range = REG_SET_FIELD(doorbell_range,
>> BIF_SDMA0_DOORBELL_RANGE, SIZE, 4);
>>   } else
>>   doorbell_range = REG_SET_FIELD(doorbell_range,
>> BIF_SDMA0_DOORBELL_RANGE, SIZE, 0);
>>   diff --git a/drivers/gpu/drm/amd/amdgpu/nbio_v7_4.c
>> b/drivers/gpu/drm/amd/amdgpu/nbio_v7_4.c
>> index f8cee95d61cc..9342ee03d7d4 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/nbio_v7_4.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/nbio_v7_4.c
>> @@ -76,7 +76,7 @@ static void nbio_v7_4_sdma_doorbell_range(struct
>> amdgpu_device *adev, int instan
>>     if (use_doorbell) {
>>   doorbell_range = REG_SET_FIELD(doorbell_range,
>> BIF_SDMA0_DOORBELL_RANGE, OFFSET, doorbell_index);
>> -    doorbell_range = REG_SET_FIELD(doorbell_range,
>> BIF_SDMA0_DOORBELL_RANGE, SIZE, 2);
>> +    doorbell_range = REG_SET_FIELD(doorbell_range,
>> BIF_SDMA0_DOORBELL_RANGE, SIZE, 4);
>>   } else
>>   doorbell_range = REG_SET_FIELD(doorbell_range,
>> BIF_SDMA0_DOORBELL_RANGE, SIZE, 0);
>>   diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c
>> b/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c
>> index f4490cdd9804..1f19f15bb171 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c
>> @@ -925,11 +925,9 @@ static void sdma_v4_0_page_resume(struct
>> amdgpu_device *adev, unsigned int i)
>>   OFFSET, ring->doorbell_index);
>>   WREG32_SDMA(i, mmSDMA0_PAGE_DOORBELL, doorbell);
>>   WREG32_SDMA(i, mmSDMA0_PAGE_DOORBELL_OFFSET, doorbell_offset);
>> -    /* TODO: enable doorbell support */
>> -    /*adev->nbio_funcs->sdma_doorbell_range(adev, i,
>> ring->use_doorbell,
>> -  ring->doorbell_index);*/
>>   -    sdma_v4_0_ring_set_wptr(ring);
>> +    /* paging queue doorbell index is already enabled at
>> sdma_v4_0_gfx_resume */
>> +    sdma_v4_0_page_ring_set_wptr(ring);
>>     /* set minor_ptr_update to 0 after wptr programed */
>>   WREG32_SDMA(i, mmSDMA0_PAGE_MINOR_PTR_UPDATE, 0);
>> @@ -1504,9 +1502,6 @@ static int sdma_v4_0_sw_init(void *handle)
>>   ring->ring_obj = NULL;
>>   ring->use_doorbell = true;
>>   -    DRM_INFO("use_doorbell being set to: [%s]\n",
>> -    ring->use_doorbell?"true":"false");
>> -
>>   if (adev->asic_type == CHIP_VEGA10)
>>   ring->doorbell_index = (i == 0) ?
>>   (AMDGPU_VEGA10_DOORBELL64_sDMA_ENGINE0 << 1) //get
>> DWORD offset
>> @@ -1516,6 +1511,8 @@ static int sdma_v4_0_sw_init(void *handle)
>>   (AMDGPU_DOORBELL64_sDMA_ENGINE0 << 1) //get DWORD
>> offset
>>   : (AMDGPU_DOORBELL64_sDMA_ENGINE1 << 1); // get
>> DWORD offset
>>   +    DRM_DEBUG("use_doorbell being set to: [%s] doorbell index
>> %d\n",
>> +    ring->use_doorbell?"true":"false", ring->doorbell_index);
>>     sprintf(ring->name, "sdma%d", i);
>>   r = amdgpu_ring_init(adev, ring, 1024,
>> @@ -1529,7 +1526,20 @@ static int sdma_v4_0_sw_init(void *handle)
>>   if (adev->sdma.has_page_queue) {
>>   ring = &adev->sdma.instance[i].page;
>>   ring->ring_obj = NULL;
>> -    ring->use_doorbell = false;
>> +    if (!amdgpu_sriov_vf(adev) || adev->asic_type ==
>> CHIP_VEGA20)
>> +    ring->use_doorbell = true;
>> +
>> +    if (adev->asi

[PATCH 2/4] drm/amdkfd: Add NULL-pointer check

[Kuehling, Felix]

top_dev->gpu is NULL for CPUs. Avoid dereferencing it if NULL.

Signed-off-by: Felix Kuehling 
---
 drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c 
b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
index aa793fc..c5ed21e 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
@@ -101,7 +101,7 @@ struct kfd_dev *kfd_device_by_pci_dev(const struct pci_dev 
*pdev)
down_read(&topology_lock);
 
list_for_each_entry(top_dev, &topology_device_list, list)
-   if (top_dev->gpu->pdev == pdev) {
+   if (top_dev->gpu && top_dev->gpu->pdev == pdev) {
device = top_dev->gpu;
break;
}
-- 
2.7.4

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 0/4] KFD upstreaming Nov 2018, part 2

[Kuehling, Felix]

This round adds support for more ROCm memory manager features:
* VRAM limit checking to avoid overcommitment
* DMABuf import for graphics interoperability
* Support for mapping doorbells into GPUVM address space

Felix Kuehling (4):
  drm/amdgpu: Add KFD VRAM limit checking
  drm/amdkfd: Add NULL-pointer check
  drm/amdkfd: Add DMABuf import functionality
  drm/amdkfd: Add support for doorbell BOs

 drivers/gpu/drm/amd/amdgpu/amdgpu.h  |   7 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c   | 109 ---
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h   |  19 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 170 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h  |   2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c   |   2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_prime.c|   4 +-
 drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 124 -
 drivers/gpu/drm/amd/amdkfd/kfd_priv.h|   1 +
 drivers/gpu/drm/amd/amdkfd/kfd_topology.c|  20 ++-
 drivers/gpu/drm/amd/include/kgd_kfd_interface.h  |   4 +-
 include/uapi/linux/kfd_ioctl.h   |  26 +++-
 12 files changed, 423 insertions(+), 65 deletions(-)

-- 
2.7.4

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 3/4] drm/amdkfd: Add DMABuf import functionality

[Kuehling, Felix]

This is used for interoperability between ROCm compute and graphics
APIs. It allows importing graphics driver BOs into the ROCm SVM
address space for zero-copy GPU access.

The API is split into two steps (query and import) to allow user mode
to manage the virtual address space allocation for the imported buffer.

Signed-off-by: Felix Kuehling 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c   |  57 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h   |  11 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c |  55 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h  |   2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_prime.c|   4 +-
 drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 118 ++-
 drivers/gpu/drm/amd/amdkfd/kfd_priv.h|   1 +
 drivers/gpu/drm/amd/amdkfd/kfd_topology.c|  18 
 include/uapi/linux/kfd_ioctl.h   |  26 -
 9 files changed, 287 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
index 274099a..44b8a12 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
@@ -26,6 +26,7 @@
 #include "amdgpu.h"
 #include "amdgpu_gfx.h"
 #include 
+#include 
 
 const struct kgd2kfd_calls *kgd2kfd;
 
@@ -444,6 +445,62 @@ void amdgpu_amdkfd_get_cu_info(struct kgd_dev *kgd, struct 
kfd_cu_info *cu_info)
cu_info->lds_size = acu_info.lds_size;
 }
 
+int amdgpu_amdkfd_get_dmabuf_info(struct kgd_dev *kgd, int dma_buf_fd,
+ struct kgd_dev **dma_buf_kgd,
+ uint64_t *bo_size, void *metadata_buffer,
+ size_t buffer_size, uint32_t *metadata_size,
+ uint32_t *flags)
+{
+   struct amdgpu_device *adev = (struct amdgpu_device *)kgd;
+   struct dma_buf *dma_buf;
+   struct drm_gem_object *obj;
+   struct amdgpu_bo *bo;
+   uint64_t metadata_flags;
+   int r = -EINVAL;
+
+   dma_buf = dma_buf_get(dma_buf_fd);
+   if (IS_ERR(dma_buf))
+   return PTR_ERR(dma_buf);
+
+   if (dma_buf->ops != &amdgpu_dmabuf_ops)
+   /* Can't handle non-graphics buffers */
+   goto out_put;
+
+   obj = dma_buf->priv;
+   if (obj->dev->driver != adev->ddev->driver)
+   /* Can't handle buffers from different drivers */
+   goto out_put;
+
+   adev = obj->dev->dev_private;
+   bo = gem_to_amdgpu_bo(obj);
+   if (!(bo->preferred_domains & (AMDGPU_GEM_DOMAIN_VRAM |
+   AMDGPU_GEM_DOMAIN_GTT)))
+   /* Only VRAM and GTT BOs are supported */
+   goto out_put;
+
+   r = 0;
+   if (dma_buf_kgd)
+   *dma_buf_kgd = (struct kgd_dev *)adev;
+   if (bo_size)
+   *bo_size = amdgpu_bo_size(bo);
+   if (metadata_size)
+   *metadata_size = bo->metadata_size;
+   if (metadata_buffer)
+   r = amdgpu_bo_get_metadata(bo, metadata_buffer, buffer_size,
+  metadata_size, &metadata_flags);
+   if (flags) {
+   *flags = (bo->preferred_domains & AMDGPU_GEM_DOMAIN_VRAM) ?
+   ALLOC_MEM_FLAGS_VRAM : ALLOC_MEM_FLAGS_GTT;
+
+   if (bo->flags & AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED)
+   *flags |= ALLOC_MEM_FLAGS_PUBLIC;
+   }
+
+out_put:
+   dma_buf_put(dma_buf);
+   return r;
+}
+
 uint64_t amdgpu_amdkfd_get_vram_usage(struct kgd_dev *kgd)
 {
struct amdgpu_device *adev = (struct amdgpu_device *)kgd;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
index 4d766cb..1a84fe2 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
@@ -150,6 +150,11 @@ uint64_t amdgpu_amdkfd_get_gpu_clock_counter(struct 
kgd_dev *kgd);
 
 uint32_t amdgpu_amdkfd_get_max_engine_clock_in_mhz(struct kgd_dev *kgd);
 void amdgpu_amdkfd_get_cu_info(struct kgd_dev *kgd, struct kfd_cu_info 
*cu_info);
+int amdgpu_amdkfd_get_dmabuf_info(struct kgd_dev *kgd, int dma_buf_fd,
+ struct kgd_dev **dmabuf_kgd,
+ uint64_t *bo_size, void *metadata_buffer,
+ size_t buffer_size, uint32_t *metadata_size,
+ uint32_t *flags);
 uint64_t amdgpu_amdkfd_get_vram_usage(struct kgd_dev *kgd);
 uint64_t amdgpu_amdkfd_get_hive_id(struct kgd_dev *kgd);
 
@@ -201,6 +206,12 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void 
*process_info,
 int amdgpu_amdkfd_gpuvm_get_vm_fault_info(struct kgd_dev *kgd,
  struct kfd_vm_fault_info *info);
 
+int amdgpu_amdkfd_gpuvm_import_dmabuf(struct kgd_dev *kgd,
+ struct dma_buf *dmabuf,

[PATCH 1/4] drm/amdgpu: Add KFD VRAM limit checking

[Kuehling, Felix]

We don't want KFD processes evicting each other over VRAM usage.
Therefore prevent overcommitting VRAM among KFD applications with
a per-GPU limit. Also leave enough room for page tables on top
of the application memory usage.

Signed-off-by: Felix Kuehling 
Reviewed-by: Eric Huang 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h  |  7 +--
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c   | 52 
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h   |  8 +++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 60 
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c   |  2 +-
 5 files changed, 75 insertions(+), 54 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index 2c80453..40e084a9 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -81,6 +81,7 @@
 #include "amdgpu_job.h"
 #include "amdgpu_bo_list.h"
 #include "amdgpu_gem.h"
+#include "amdgpu_amdkfd.h"
 
 #define MAX_GPU_INSTANCE   16
 
@@ -979,6 +980,9 @@ struct amdgpu_device {
/* GDS */
struct amdgpu_gds   gds;
 
+   /* KFD */
+   struct amdgpu_kfd_dev   kfd;
+
/* display related functionality */
struct amdgpu_display_manager dm;
 
@@ -992,9 +996,6 @@ struct amdgpu_device {
atomic64_t visible_pin_size;
atomic64_t gart_pin_size;
 
-   /* amdkfd interface */
-   struct kfd_dev  *kfd;
-
/* soc15 register offset based on ip, instance and  segment */
uint32_t*reg_offset[MAX_HWIP][HWIP_MAX_INSTANCE];
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
index 1c1fed6..274099a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
@@ -31,10 +31,20 @@ const struct kgd2kfd_calls *kgd2kfd;
 
 static const unsigned int compute_vmid_bitmap = 0xFF00;
 
+/* Total memory size in system memory and all GPU VRAM. Used to
+ * estimate worst case amount of memory to reserve for page tables
+ */
+uint64_t amdgpu_amdkfd_total_mem_size;
+
 int amdgpu_amdkfd_init(void)
 {
+   struct sysinfo si;
int ret;
 
+   si_meminfo(&si);
+   amdgpu_amdkfd_total_mem_size = si.totalram - si.totalhigh;
+   amdgpu_amdkfd_total_mem_size *= si.mem_unit;
+
 #ifdef CONFIG_HSA_AMD
ret = kgd2kfd_init(KFD_INTERFACE_VERSION, &kgd2kfd);
if (ret)
@@ -87,8 +97,11 @@ void amdgpu_amdkfd_device_probe(struct amdgpu_device *adev)
return;
}
 
-   adev->kfd = kgd2kfd->probe((struct kgd_dev *)adev,
-  adev->pdev, kfd2kgd);
+   adev->kfd.dev = kgd2kfd->probe((struct kgd_dev *)adev,
+  adev->pdev, kfd2kgd);
+
+   if (adev->kfd.dev)
+   amdgpu_amdkfd_total_mem_size += adev->gmc.real_vram_size;
 }
 
 /**
@@ -128,7 +141,8 @@ void amdgpu_amdkfd_device_init(struct amdgpu_device *adev)
 {
int i, n;
int last_valid_bit;
-   if (adev->kfd) {
+
+   if (adev->kfd.dev) {
struct kgd2kfd_shared_resources gpu_resources = {
.compute_vmid_bitmap = compute_vmid_bitmap,
.num_pipe_per_mec = adev->gfx.mec.num_pipe_per_mec,
@@ -167,7 +181,7 @@ void amdgpu_amdkfd_device_init(struct amdgpu_device *adev)
&gpu_resources.doorbell_start_offset);
 
if (adev->asic_type < CHIP_VEGA10) {
-   kgd2kfd->device_init(adev->kfd, &gpu_resources);
+   kgd2kfd->device_init(adev->kfd.dev, &gpu_resources);
return;
}
 
@@ -207,37 +221,37 @@ void amdgpu_amdkfd_device_init(struct amdgpu_device *adev)
gpu_resources.reserved_doorbell_mask = 0x1e0;
gpu_resources.reserved_doorbell_val  = 0x0e0;
 
-   kgd2kfd->device_init(adev->kfd, &gpu_resources);
+   kgd2kfd->device_init(adev->kfd.dev, &gpu_resources);
}
 }
 
 void amdgpu_amdkfd_device_fini(struct amdgpu_device *adev)
 {
-   if (adev->kfd) {
-   kgd2kfd->device_exit(adev->kfd);
-   adev->kfd = NULL;
+   if (adev->kfd.dev) {
+   kgd2kfd->device_exit(adev->kfd.dev);
+   adev->kfd.dev = NULL;
}
 }
 
 void amdgpu_amdkfd_interrupt(struct amdgpu_device *adev,
const void *ih_ring_entry)
 {
-   if (adev->kfd)
-   kgd2kfd->interrupt(adev->kfd, ih_ring_entry);
+   if (adev->kfd.dev)
+   kgd2kfd->interrupt(adev->kfd.dev, ih_ring_entry);
 }
 
 void amdgpu_amdkfd_suspend(struct amdgpu_device *adev)
 {
-   if (adev->kfd)
-   kgd2kfd->suspend(adev->kfd);
+   if (adev->kfd.dev)
+   kgd2kfd->suspend(adev->kfd.dev);
 }
 
 int amdgpu_amdkfd_resume(struct amdgpu_device *adev)
 {
int r = 0

[PATCH 4/4] drm/amdkfd: Add support for doorbell BOs

[Kuehling, Felix]

This allows user mode to map doorbell pages into GPUVM address space.
That way GPUs can submit to user mode queues (self-dispatch).

Signed-off-by: Felix Kuehling 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 59 ++--
 drivers/gpu/drm/amd/amdkfd/kfd_chardev.c |  6 +++
 drivers/gpu/drm/amd/include/kgd_kfd_interface.h  |  4 +-
 3 files changed, 62 insertions(+), 7 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
index 14f76765..f3ac8e5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
@@ -887,6 +887,24 @@ static int map_bo_to_gpuvm(struct amdgpu_device *adev,
return ret;
 }
 
+static struct sg_table *create_doorbell_sg(uint64_t addr, uint32_t size)
+{
+   struct sg_table *sg = kmalloc(sizeof(*sg), GFP_KERNEL);
+
+   if (!sg)
+   return NULL;
+   if (sg_alloc_table(sg, 1, GFP_KERNEL)) {
+   kfree(sg);
+   return NULL;
+   }
+   sg->sgl->dma_address = addr;
+   sg->sgl->length = size;
+#ifdef CONFIG_NEED_SG_DMA_LENGTH
+   sg->sgl->dma_length = size;
+#endif
+   return sg;
+}
+
 static int process_validate_vms(struct amdkfd_process_info *process_info)
 {
struct amdgpu_vm *peer_vm;
@@ -1170,6 +1188,8 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(
 {
struct amdgpu_device *adev = get_amdgpu_device(kgd);
struct amdgpu_vm *avm = (struct amdgpu_vm *)vm;
+   enum ttm_bo_type bo_type = ttm_bo_type_device;
+   struct sg_table *sg = NULL;
uint64_t user_addr = 0;
struct amdgpu_bo *bo;
struct amdgpu_bo_param bp;
@@ -1198,13 +1218,25 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(
if (!offset || !*offset)
return -EINVAL;
user_addr = *offset;
+   } else if (flags & ALLOC_MEM_FLAGS_DOORBELL) {
+   domain = AMDGPU_GEM_DOMAIN_GTT;
+   alloc_domain = AMDGPU_GEM_DOMAIN_CPU;
+   bo_type = ttm_bo_type_sg;
+   alloc_flags = 0;
+   if (size > UINT_MAX)
+   return -EINVAL;
+   sg = create_doorbell_sg(*offset, size);
+   if (!sg)
+   return -ENOMEM;
} else {
return -EINVAL;
}
 
*mem = kzalloc(sizeof(struct kgd_mem), GFP_KERNEL);
-   if (!*mem)
-   return -ENOMEM;
+   if (!*mem) {
+   ret = -ENOMEM;
+   goto err;
+   }
INIT_LIST_HEAD(&(*mem)->bo_va_list);
mutex_init(&(*mem)->lock);
(*mem)->aql_queue = !!(flags & ALLOC_MEM_FLAGS_AQL_QUEUE_MEM);
@@ -1237,7 +1269,7 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(
 
amdgpu_sync_create(&(*mem)->sync);
 
-   ret = amdgpu_amdkfd_reserve_mem_limit(adev, size, alloc_domain, false);
+   ret = amdgpu_amdkfd_reserve_mem_limit(adev, size, alloc_domain, !!sg);
if (ret) {
pr_debug("Insufficient system memory\n");
goto err_reserve_limit;
@@ -1251,7 +1283,7 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(
bp.byte_align = byte_align;
bp.domain = alloc_domain;
bp.flags = alloc_flags;
-   bp.type = ttm_bo_type_device;
+   bp.type = bo_type;
bp.resv = NULL;
ret = amdgpu_bo_create(adev, &bp, &bo);
if (ret) {
@@ -1259,6 +1291,10 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(
domain_string(alloc_domain), ret);
goto err_bo_create;
}
+   if (bo_type == ttm_bo_type_sg) {
+   bo->tbo.sg = sg;
+   bo->tbo.ttm->sg = sg;
+   }
bo->kfd_bo = *mem;
(*mem)->bo = bo;
if (user_addr)
@@ -1290,10 +1326,15 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(
/* Don't unreserve system mem limit twice */
goto err_reserve_limit;
 err_bo_create:
-   unreserve_mem_limit(adev, size, alloc_domain, false);
+   unreserve_mem_limit(adev, size, alloc_domain, !!sg);
 err_reserve_limit:
mutex_destroy(&(*mem)->lock);
kfree(*mem);
+err:
+   if (sg) {
+   sg_free_table(sg);
+   kfree(sg);
+   }
return ret;
 }
 
@@ -1363,6 +1404,14 @@ int amdgpu_amdkfd_gpuvm_free_memory_of_gpu(
/* Free the sync object */
amdgpu_sync_free(&mem->sync);
 
+   /* If the SG is not NULL, it's one we created for a doorbell
+* BO. We need to free it.
+*/
+   if (mem->bo->tbo.sg) {
+   sg_free_table(mem->bo->tbo.sg);
+   kfree(mem->bo->tbo.sg);
+   }
+
/* Free the BO*/
amdgpu_bo_unref(&mem->bo);
mutex_destroy(&mem->lock);
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c 
b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
index ae3ae0f..3623538 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_

Re: [PATCH] drm/amdgpu: Add delay after enable RLC ucode

[Kuehling, Felix]

On 2018-11-22 12:03 p.m., Liu, Shaoyun wrote:
> Driver shouldn't try to access any GFX registers until RLC is idle.
> During the test, it took 12 seconds for RLC to clear the BUSY bit
> in RLC_GPM_STAT register which is un-acceptable for driver.
> As per RLC engineer, it would take RLC Ucode less than 10,000 GFXCLK
> cycles to finish its critical section. In a lowest 300M enginer clock
> setting(default from vbios), 50 us delay is enough.
>
> Change-Id: If074fbeeecad3f63cf5f5ab232472d0fe46082a6
> Signed-off-by: shaoyunl 
> ---
>  drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c 
> b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
> index e409fbf..2e3ee42 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
> @@ -2326,6 +2326,7 @@ static void gfx_v9_0_rlc_start(struct amdgpu_device 
> *adev)
>  #endif
>  
>   WREG32_FIELD15(GC, 0, RLC_CNTL, RLC_ENABLE_F32, 1);
> + udelay(50);
>  
>   /* carrizo do enable cp interrupt after cp inited */
>   if (!(adev->flags & AMD_IS_APU))

gfx_v9_0_enable_gui_idle_interrupt(adev, true);

udelay(50);

There is already another udelay(50) here. Is that one still needed?

Regards,
  Felix

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] mm: convert totalram_pages, totalhigh_pages and managed_pages to atomic.

[Kuehling, Felix]

On 2018-10-22 1:23 p.m., Arun KS wrote:
> Remove managed_page_count_lock spinlock and instead use atomic
> variables.
>
> Suggested-by: Michal Hocko 
> Suggested-by: Vlastimil Babka 
> Signed-off-by: Arun KS 

Acked-by: Felix Kuehling 

Regards,
  Felix

>
> ---
> As discussed here,
> https://patchwork.kernel.org/patch/10627521/#22261253
> ---
> ---
>  arch/csky/mm/init.c   |  4 +-
>  arch/powerpc/platforms/pseries/cmm.c  | 11 ++--
>  arch/s390/mm/init.c   |  2 +-
>  arch/um/kernel/mem.c  |  4 +-
>  arch/x86/kernel/cpu/microcode/core.c  |  5 +-
>  drivers/char/agp/backend.c|  4 +-
>  drivers/gpu/drm/amd/amdkfd/kfd_crat.c |  2 +-
>  drivers/gpu/drm/i915/i915_gem.c   |  2 +-
>  drivers/gpu/drm/i915/selftests/i915_gem_gtt.c |  4 +-
>  drivers/hv/hv_balloon.c   | 19 +++
>  drivers/md/dm-bufio.c |  5 +-
>  drivers/md/dm-crypt.c |  4 +-
>  drivers/md/dm-integrity.c |  4 +-
>  drivers/md/dm-stats.c |  3 +-
>  drivers/media/platform/mtk-vpu/mtk_vpu.c  |  3 +-
>  drivers/misc/vmw_balloon.c|  2 +-
>  drivers/parisc/ccio-dma.c |  5 +-
>  drivers/parisc/sba_iommu.c|  5 +-
>  drivers/staging/android/ion/ion_system_heap.c |  2 +-
>  drivers/xen/xen-selfballoon.c |  7 +--
>  fs/ceph/super.h   |  3 +-
>  fs/file_table.c   |  9 ++--
>  fs/fuse/inode.c   |  4 +-
>  fs/nfs/write.c|  3 +-
>  fs/nfsd/nfscache.c|  3 +-
>  fs/ntfs/malloc.h  |  2 +-
>  fs/proc/base.c|  3 +-
>  include/linux/highmem.h   |  2 +-
>  include/linux/mm.h|  2 +-
>  include/linux/mmzone.h| 10 +---
>  include/linux/swap.h  |  2 +-
>  kernel/fork.c |  6 +--
>  kernel/kexec_core.c   |  5 +-
>  kernel/power/snapshot.c   |  2 +-
>  lib/show_mem.c|  3 +-
>  mm/highmem.c  |  2 +-
>  mm/huge_memory.c  |  2 +-
>  mm/kasan/quarantine.c |  4 +-
>  mm/memblock.c |  6 +--
>  mm/memory_hotplug.c   |  4 +-
>  mm/mm_init.c  |  3 +-
>  mm/oom_kill.c |  2 +-
>  mm/page_alloc.c   | 75 
> ++-
>  mm/shmem.c| 12 +++--
>  mm/slab.c |  3 +-
>  mm/swap.c |  3 +-
>  mm/util.c |  2 +-
>  mm/vmalloc.c  |  4 +-
>  mm/vmstat.c   |  4 +-
>  mm/workingset.c   |  2 +-
>  mm/zswap.c|  2 +-
>  net/dccp/proto.c  |  6 +--
>  net/decnet/dn_route.c |  2 +-
>  net/ipv4/tcp_metrics.c|  2 +-
>  net/netfilter/nf_conntrack_core.c |  6 +--
>  net/netfilter/xt_hashlimit.c  |  4 +-
>  net/sctp/protocol.c   |  6 +--
>  security/integrity/ima/ima_kexec.c|  2 +-
>  58 files changed, 171 insertions(+), 143 deletions(-)
>
> diff --git a/arch/csky/mm/init.c b/arch/csky/mm/init.c
> index dc07c07..3f4d35e 100644
> --- a/arch/csky/mm/init.c
> +++ b/arch/csky/mm/init.c
> @@ -71,7 +71,7 @@ void free_initrd_mem(unsigned long start, unsigned long end)
>   ClearPageReserved(virt_to_page(start));
>   init_page_count(virt_to_page(start));
>   free_page(start);
> - totalram_pages++;
> + atomic_long_inc(&totalram_pages);
>   }
>  }
>  #endif
> @@ -88,7 +88,7 @@ void free_initmem(void)
>   ClearPageReserved(virt_to_page(addr));
>   init_page_count(virt_to_page(addr));
>   free_page(addr);
> - totalram_pages++;
> + atomic_long_inc(&totalram_pages);
>   addr += PAGE_SIZE;
>   }
>  
> diff --git a/arch/powerpc/platforms/pseries/cmm.c 
> b/arch/powerpc/platforms/pseries/cmm.c
> index 25427a4..85fe503 100644
> --- a/arch/powerpc/platforms/pseries/cmm.c
> +++ b/arch/powerpc/platforms/pseries/cmm.c
> @@ -208,7 +208,7 @@ static long cmm_alloc_pages(long nr)
>  
>   pa->page[pa->index++] = addr;
>   loaned_pages++;
> - totalram_pages--;
> + atomic_long_dec(&totalram_page

Re: [PATCH] drm/amdgpu: Add delay after enable RLC ucode

[Kuehling, Felix]

On 2018-11-22 1:22 p.m., Liu, Shaoyun wrote:
> Driver shouldn't try to access any GFX registers until RLC is idle.
> During the test, it took 12 seconds for RLC to clear the BUSY bit
> in RLC_GPM_STAT register which is un-acceptable for driver.
> As per RLC engineer, it would take RLC Ucode less than 10,000 GFXCLK
> cycles to finish its critical section. In a lowest 300M enginer clock
> setting(default from vbios), 50 us delay is enough.
>
> Change-Id: If074fbeeecad3f63cf5f5ab232472d0fe46082a6
> Signed-off-by: shaoyunl 

See the comment from Ernst Sjöstrand. Other than that, this patch is
Acked-by: Felix Kuehling 

Maybe someone more familiar with the RLC can give a R-B.

Regards,
  Felix


> ---
>  drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 7 ---
>  1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c 
> b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
> index e409fbf..9686fcc 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
> @@ -2326,12 +2326,13 @@ static void gfx_v9_0_rlc_start(struct amdgpu_device 
> *adev)
>  #endif
>  
>   WREG32_FIELD15(GC, 0, RLC_CNTL, RLC_ENABLE_F32, 1);
> + udelay(50);
>  
>   /* carrizo do enable cp interrupt after cp inited */
> - if (!(adev->flags & AMD_IS_APU))
> + if (!(adev->flags & AMD_IS_APU)) {
>   gfx_v9_0_enable_gui_idle_interrupt(adev, true);
> -
> - udelay(50);
> + udelay(50);
> + }
>  
>  #ifdef AMDGPU_RLC_DEBUG_RETRY
>   /* RLC_GPM_GENERAL_6 : RLC Ucode version */
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 2/2] drm/amdgpu: Avoid endless loop in GPUVM fragment processing

[Kuehling, Felix]

Don't bounce back to the root level for fragment processing, because
huge pages are not supported at that level. This is unlikely to happen
with the default VM size on Vega, but can be exposed by limiting the
VM size with the amdgpu.vm_size module parameter.

Signed-off-by: Felix Kuehling 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index be3e360..0877ff9 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -1656,9 +1656,11 @@ static int amdgpu_vm_update_ptes(struct 
amdgpu_pte_update_params *params,
if (!amdgpu_vm_pt_descendant(adev, &cursor))
return -ENOENT;
continue;
-   } else if (frag >= parent_shift) {
+   } else if (frag >= parent_shift &&
+  cursor.level - 1 != adev->vm_manager.root_level) {
/* If the fragment size is even larger than the parent
-* shift we should go up one level and check it again.
+* shift we should go up one level and check it again
+* unless one level up is the root level.
 */
if (!amdgpu_vm_pt_ancestor(&cursor))
return -ENOENT;
-- 
2.7.4

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 1/2] drm/amdgpu: Cast to uint64_t before left shift

[Kuehling, Felix]

Avoid potential integer overflows with left shift in huge-page mapping
code by casting the operand to uin64_t first.

Signed-off-by: Felix Kuehling 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index dad0e23..be3e360 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -181,7 +181,7 @@ static unsigned amdgpu_vm_num_entries(struct amdgpu_device 
*adev,
 
if (level == adev->vm_manager.root_level)
/* For the root directory */
-   return round_up(adev->vm_manager.max_pfn, 1 << shift) >> shift;
+   return round_up(adev->vm_manager.max_pfn, 1ULL << shift) >> 
shift;
else if (level != AMDGPU_VM_PTB)
/* Everything in between */
return 512;
@@ -1666,10 +1666,10 @@ static int amdgpu_vm_update_ptes(struct 
amdgpu_pte_update_params *params,
}
 
/* Looks good so far, calculate parameters for the update */
-   incr = AMDGPU_GPU_PAGE_SIZE << shift;
+   incr = (uint64_t)AMDGPU_GPU_PAGE_SIZE << shift;
mask = amdgpu_vm_entries_mask(adev, cursor.level);
pe_start = ((cursor.pfn >> shift) & mask) * 8;
-   entry_end = (mask + 1) << shift;
+   entry_end = (uint64_t)(mask + 1) << shift;
entry_end += cursor.pfn & ~(entry_end - 1);
entry_end = min(entry_end, end);
 
@@ -1682,7 +1682,7 @@ static int amdgpu_vm_update_ptes(struct 
amdgpu_pte_update_params *params,
  flags | AMDGPU_PTE_FRAG(frag));
 
pe_start += nptes * 8;
-   dst += nptes * AMDGPU_GPU_PAGE_SIZE << shift;
+   dst += (uint64_t)nptes * AMDGPU_GPU_PAGE_SIZE << shift;
 
frag_start = upd_end;
if (frag_start >= frag_end) {
-- 
2.7.4

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 02/11] drm/amdgpu: send IVs to the KFD only after processing them v2

[Kuehling, Felix]

Won't this break VM fault handling in KFD? I don't see a way with the current 
code that you can leave some VM faults for KFD to process. If we could consider 
VM faults with VMIDs 8-15 as not handled in amdgpu and leave them for KFD to 
process, then this could work.

As far as I can tell, the only code path that leave IRQs unhandled and passes 
them to KFD prints an error message in the kernel log. We can't have the kernel 
log flooded with error messages every time there are IRQs for KFD. We can get 
extremely high frequency interrupts for HSA signals.

Regards,
  Felix

-Original Message-
From: amd-gfx  On Behalf Of Alex Deucher
Sent: Friday, November 30, 2018 10:03 AM
To: Christian König 
Cc: amd-gfx list 
Subject: Re: [PATCH 02/11] drm/amdgpu: send IVs to the KFD only after 
processing them v2

On Fri, Nov 30, 2018 at 7:36 AM Christian König 
 wrote:
>
> This allows us to filter out VM faults in the GMC code.
>
> v2: don't filter out all faults
>
> Signed-off-by: Christian König 

Acked-by: Alex Deucher 

> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 29 
> +++--
>  1 file changed, 17 insertions(+), 12 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c
> index 6b6524f04ce0..6db4c58ddc13 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c
> @@ -149,9 +149,6 @@ static void amdgpu_irq_callback(struct amdgpu_device 
> *adev,
> if (!amdgpu_ih_prescreen_iv(adev))
> return;
>
> -   /* Before dispatching irq to IP blocks, send it to amdkfd */
> -   amdgpu_amdkfd_interrupt(adev, (const void *) &ih->ring[ring_index]);
> -
> entry.iv_entry = (const uint32_t *)&ih->ring[ring_index];
> amdgpu_ih_decode_iv(adev, &entry);
>
> @@ -371,29 +368,31 @@ void amdgpu_irq_dispatch(struct amdgpu_device *adev,
> unsigned client_id = entry->client_id;
> unsigned src_id = entry->src_id;
> struct amdgpu_irq_src *src;
> +   bool handled = false;
> int r;
>
> trace_amdgpu_iv(entry);
>
> if (client_id >= AMDGPU_IRQ_CLIENTID_MAX) {
> -   DRM_DEBUG("Invalid client_id in IV: %d\n", client_id);
> +   DRM_ERROR("Invalid client_id in IV: %d\n", client_id);
> return;
> }
>
> if (src_id >= AMDGPU_MAX_IRQ_SRC_ID) {
> -   DRM_DEBUG("Invalid src_id in IV: %d\n", src_id);
> +   DRM_ERROR("Invalid src_id in IV: %d\n", src_id);
> return;
> }
>
> if (adev->irq.virq[src_id]) {
> generic_handle_irq(irq_find_mapping(adev->irq.domain, 
> src_id));
> -   } else {
> -   if (!adev->irq.client[client_id].sources) {
> -   DRM_DEBUG("Unregistered interrupt client_id: %d 
> src_id: %d\n",
> - client_id, src_id);
> -   return;
> -   }
> +   return;
> +   }
>
> +   if (!adev->irq.client[client_id].sources) {
> +   DRM_DEBUG("Unregistered interrupt client_id: %d src_id: %d\n",
> + client_id, src_id);
> +   return;
> +   } else {
> src = adev->irq.client[client_id].sources[src_id];
> if (!src) {
> DRM_DEBUG("Unhandled interrupt src_id: %d\n", 
> src_id); @@ -401,9 +400,15 @@ void amdgpu_irq_dispatch(struct amdgpu_device 
> *adev,
> }
>
> r = src->funcs->process(adev, src, entry);
> -   if (r)
> +   if (r < 0)
> DRM_ERROR("error processing interrupt (%d)\n", 
> r);
> +   else if (r)
> +   handled = true;
> }
> +
> +   /* Send it to amdkfd as well if it isn't already handled */
> +   if (!handled)
> +   amdgpu_amdkfd_interrupt(adev, entry->iv_entry);
>  }
>
>  /**
> --
> 2.17.1
>
> ___
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 02/11] drm/amdgpu: send IVs to the KFD only after processing them v2

[Kuehling, Felix]

On 2018-12-01 9:11 a.m., Christian König wrote:
>> Won't this break VM fault handling in KFD?
> No, we still send all VM faults to KFD after processing them. Only
> filtered retries are not send to the KFD any more.

OK, I missed that src->funcs->process returning 0 means "not handled",
>0 means "handled". Currently I don't see any interrupt processing
callbacks returning >0. I think that gets added in patch 4.


>
>> As far as I can tell, the only code path that leave IRQs unhandled
>> and passes them to KFD prints an error message in the kernel log. We
>> can't have the kernel log flooded with error messages every time
>> there are IRQs for KFD. We can get extremely high frequency
>> interrupts for HSA signals.
> Since the KFD didn't filtered the faults this would have a been a
> problem before as well.

I missed that r == 0 means not handled without being an error.


>
> So I'm pretty sure that we already have registered handlers for all
> interrupts the KFD is interested in as well.

No. As far as I can tell, you're missing these two:

GFX_9_0__SRCID__CP_BAD_OPCODE_ERROR (183)
GFX_9_0__SRCID__SQ_INTERRUPT_ID (239)

239 is used for signaling events from shaders and can be very frequent.
Triggering an error message on those interrupts would be bad.

Regards,
  Felix


>
> Regards,
> Christian.
>
> Am 30.11.18 um 17:31 schrieb Kuehling, Felix:
>> Won't this break VM fault handling in KFD? I don't see a way with the
>> current code that you can leave some VM faults for KFD to process. If
>> we could consider VM faults with VMIDs 8-15 as not handled in amdgpu
>> and leave them for KFD to process, then this could work.
>>
>> As far as I can tell, the only code path that leave IRQs unhandled
>> and passes them to KFD prints an error message in the kernel log. We
>> can't have the kernel log flooded with error messages every time
>> there are IRQs for KFD. We can get extremely high frequency
>> interrupts for HSA signals.
>>
>> Regards,
>>    Felix
>>
>> -Original Message-
>> From: amd-gfx  On Behalf Of
>> Alex Deucher
>> Sent: Friday, November 30, 2018 10:03 AM
>> To: Christian König 
>> Cc: amd-gfx list 
>> Subject: Re: [PATCH 02/11] drm/amdgpu: send IVs to the KFD only after
>> processing them v2
>>
>> On Fri, Nov 30, 2018 at 7:36 AM Christian König
>>  wrote:
>>> This allows us to filter out VM faults in the GMC code.
>>>
>>> v2: don't filter out all faults
>>>
>>> Signed-off-by: Christian König 
>> Acked-by: Alex Deucher 
>>
>>> ---
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 29
>>> +++--
>>>   1 file changed, 17 insertions(+), 12 deletions(-)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c
>>> index 6b6524f04ce0..6db4c58ddc13 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c
>>> @@ -149,9 +149,6 @@ static void amdgpu_irq_callback(struct
>>> amdgpu_device *adev,
>>>  if (!amdgpu_ih_prescreen_iv(adev))
>>>  return;
>>>
>>> -   /* Before dispatching irq to IP blocks, send it to amdkfd */
>>> -   amdgpu_amdkfd_interrupt(adev, (const void *)
>>> &ih->ring[ring_index]);
>>> -
>>>  entry.iv_entry = (const uint32_t *)&ih->ring[ring_index];
>>>  amdgpu_ih_decode_iv(adev, &entry);
>>>
>>> @@ -371,29 +368,31 @@ void amdgpu_irq_dispatch(struct amdgpu_device
>>> *adev,
>>>  unsigned client_id = entry->client_id;
>>>  unsigned src_id = entry->src_id;
>>>  struct amdgpu_irq_src *src;
>>> +   bool handled = false;
>>>  int r;
>>>
>>>  trace_amdgpu_iv(entry);
>>>
>>>  if (client_id >= AMDGPU_IRQ_CLIENTID_MAX) {
>>> -   DRM_DEBUG("Invalid client_id in IV: %d\n", client_id);
>>> +   DRM_ERROR("Invalid client_id in IV: %d\n", client_id);
>>>  return;
>>>  }
>>>
>>>  if (src_id >= AMDGPU_MAX_IRQ_SRC_ID) {
>>> -   DRM_DEBUG("Invalid src_id in IV: %d\n", src_id);
>>> +   DRM_ERROR("Invalid src_id in IV: %d\n", src_id);
>>>  return;
>>>  }

Re: [PATCH] drm/amdgpu: Update XGMI node print

[Kuehling, Felix]

Shaoyun, FYI


Acked-by: Felix Kuehling 


On 2018-12-03 3:28 p.m., Deucher, Alexander wrote:
>
> Acked-by: Alex Deucher 
>
> 
> *From:* amd-gfx  on behalf of
> Andrey Grodzovsky 
> *Sent:* Monday, December 3, 2018 3:03:41 PM
> *To:* amd-gfx@lists.freedesktop.org
> *Cc:* Deucher, Alexander; Grodzovsky, Andrey
> *Subject:* [PATCH] drm/amdgpu: Update XGMI node print
>  
> amdgpu_xgmi_update_topology is called both on device registration
> and reset. Fix misleading print since the device is added only once to
> the hive on registration and not on reset.
>
> Signed-off-by: Andrey Grodzovsky 
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c
> index 1b15ff3..0b263a9 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c
> @@ -78,7 +78,7 @@ int amdgpu_xgmi_update_topology(struct
> amdgpu_hive_info *hive, struct amdgpu_dev
>  adev->gmc.xgmi.node_id,
>  adev->gmc.xgmi.hive_id, ret);
>  else
> -   dev_info(adev->dev, "XGMI: Add node %d to hive 0x%llx.\n",
> +   dev_info(adev->dev, "XGMI: Set topology for node %d,
> hive 0x%llx.\n",
>   adev->gmc.xgmi.physical_node_id,
>   adev->gmc.xgmi.hive_id);
>  
> -- 
> 2.7.4
>
> ___
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
>
> ___
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [Intel-gfx] [PATCH RFC 2/5] cgroup: Add mechanism to register vendor specific DRM devices

[Kuehling, Felix]

On 2018-11-28 4:14 a.m., Joonas Lahtinen wrote:
> Quoting Ho, Kenny (2018-11-27 17:41:17)
>> On Tue, Nov 27, 2018 at 4:46 AM Joonas Lahtinen 
>>  wrote:
>>> I think a more abstract property "% of GPU (processing power)" might
>>> be a more universal approach. One can then implement that through
>>> subdividing the resources or timeslicing them, depending on the GPU
>>> topology.
>>>
>>> Leasing 1/8th, 1/4th or 1/2 of the GPU would probably be the most
>>> applicable to cloud provider usecases, too. At least that's what I
>>> see done for the CPUs today.
>> I think there are opportunities to slice the gpu in more than one way 
>> (similar to the way it is done for cpu.)  We can potentially frame resources 
>> as continuous or discrete.  Percentage definitely fits well for continuous 
>> measurements such as time/time slices but I think there are places for 
>> discrete units such as core counts as well.
> I think the ask in return to the early series from Intal was to agree
> on the variables that could be common to all of DRM subsystem.
>
> So we can only choose the lowest common denominator, right?
>
> Any core count out of total core count should translate nicely into a
> fraction, so what would be the problem with percentage amounts?
How would you handle overcommitment with a percentage? That is, more
than 100% of the GPU cores assigned to cgroups. Which cgroups end up
sharing cores would be up to chance.

If we allow specifying a set of GPU cores, we can be more specific in
assigning and sharing resources between cgroups.

Regards,
  Felix


>
> Regards, Joonas
>
>> Regards,
>> Kenny
>>
>>> That combined with the "GPU memory usable" property should be a good
>>> starting point to start subdividing the GPU resources for multiple
>>> users.
>>>
>>> Regards, Joonas
>>>
 Your feedback is highly appreciated.

 Best Regards,
 Harish



 From: amd-gfx  on behalf of Tejun 
 Heo 
 Sent: Tuesday, November 20, 2018 5:30 PM
 To: Ho, Kenny
 Cc: cgro...@vger.kernel.org; intel-...@lists.freedesktop.org; 
 y2ke...@gmail.com; amd-gfx@lists.freedesktop.org; 
 dri-de...@lists.freedesktop.org
 Subject: Re: [PATCH RFC 2/5] cgroup: Add mechanism to register vendor 
 specific DRM devices


 Hello,

 On Tue, Nov 20, 2018 at 10:21:14PM +, Ho, Kenny wrote:
> By this reply, are you suggesting that vendor specific resources
> will never be acceptable to be managed under cgroup?  Let say a user
 I wouldn't say never but whatever which gets included as a cgroup
 controller should have clearly defined resource abstractions and the
 control schemes around them including support for delegation.  AFAICS,
 gpu side still seems to have a long way to go (and it's not clear
 whether that's somewhere it will or needs to end up).

> want to have similar functionality as what cgroup is offering but to
> manage vendor specific resources, what would you suggest as a
> solution?  When you say keeping vendor specific resource regulation
> inside drm or specific drivers, do you mean we should replicate the
> cgroup infrastructure there or do you mean either drm or specific
> driver should query existing hierarchy (such as device or perhaps
> cpu) for the process organization information?
>
> To put the questions in more concrete terms, let say a user wants to
> expose certain part of a gpu to a particular cgroup similar to the
> way selective cpu cores are exposed to a cgroup via cpuset, how
> should we go about enabling such functionality?
 Do what the intel driver or bpf is doing?  It's not difficult to hook
 into cgroup for identification purposes.

 Thanks.

 --
 tejun
 ___
 amd-gfx mailing list
 amd-gfx@lists.freedesktop.org
 https://lists.freedesktop.org/mailman/listinfo/amd-gfx


 amd-gfx Info Page - freedesktop.org
 lists.freedesktop.org
 To see the collection of prior postings to the list, visit the amd-gfx 
 Archives.. Using amd-gfx: To post a message to all the list members, send 
 email to amd-gfx@lists.freedesktop.org. You can subscribe to the list, or 
 change your existing subscription, in the sections below.

 ___
 Intel-gfx mailing list
 intel-...@lists.freedesktop.org
 https://lists.freedesktop.org/mailman/listinfo/intel-gfx
> ___
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 0/4] KFD upstreaming Nov 2018, part 2

[Kuehling, Felix]

Ping. Any comments, R-b, A-b?

On 2018-11-20 10:07 p.m., Kuehling, Felix wrote:
> This round adds support for more ROCm memory manager features:
> * VRAM limit checking to avoid overcommitment
> * DMABuf import for graphics interoperability
> * Support for mapping doorbells into GPUVM address space
>
> Felix Kuehling (4):
>   drm/amdgpu: Add KFD VRAM limit checking
>   drm/amdkfd: Add NULL-pointer check
>   drm/amdkfd: Add DMABuf import functionality
>   drm/amdkfd: Add support for doorbell BOs
>
>  drivers/gpu/drm/amd/amdgpu/amdgpu.h  |   7 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c   | 109 ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h   |  19 ++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 170 
> ++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h  |   2 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c   |   2 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_prime.c|   4 +-
>  drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 124 -
>  drivers/gpu/drm/amd/amdkfd/kfd_priv.h|   1 +
>  drivers/gpu/drm/amd/amdkfd/kfd_topology.c|  20 ++-
>  drivers/gpu/drm/amd/include/kgd_kfd_interface.h  |   4 +-
>  include/uapi/linux/kfd_ioctl.h   |  26 +++-
>  12 files changed, 423 insertions(+), 65 deletions(-)
>
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 2/2] drm/amdgpu: replace get_user_pages with HMM address mirror helpers v2

[Kuehling, Felix]

See comments inline. I didn't review the amdgpu_cs and amdgpu_gem parts
as I don't know them very well.

On 2018-12-03 3:19 p.m., Yang, Philip wrote:
> Use HMM helper function hmm_vma_fault() to get physical pages backing
> userptr and start CPU page table update track of those pages. Then use
> hmm_vma_range_done() to check if those pages are updated before
> amdgpu_cs_submit for gfx or before user queues are resumed for kfd.
>
> If userptr pages are updated, for gfx, amdgpu_cs_ioctl will restart
> from scratch, for kfd, restore worker is rescheduled to retry.
>
> To avoid circular lock dependency, no nested locking between mmap_sem
> and bo::reserve. The locking order is:
> bo::reserve -> amdgpu_mn_lock(p->mn)
>
> HMM simplify the CPU page table concurrent update check, so remove
> guptasklock, mmu_invalidations, last_set_pages fields from
> amdgpu_ttm_tt struct.
>
> HMM does not pin the page (increase page ref count), so remove related
> operations like release_pages(), put_page(), mark_page_dirty().
>
> v2:
> * Remove nested locking between mmap_sem and bo::reserve
> * Change locking order to bo::reserve -> amdgpu_mn_lock()
> * Use dynamic allocation to replace VLA in kernel stack
>
> Change-Id: Iffd5f855cc9ce402cdfca167f68f83fe39ac56f9
> Signed-off-by: Philip Yang 
> ---
>  .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  | 101 --
>  drivers/gpu/drm/amd/amdgpu/amdgpu_bo_list.c   |   2 -
>  drivers/gpu/drm/amd/amdgpu/amdgpu_bo_list.h   |   3 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c| 188 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c   |  14 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c|  34 +++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h|   7 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c   | 164 ++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h   |   3 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c|   1 -
>  .../drm/amd/amdkfd/kfd_device_queue_manager.c |  67 ---
>  11 files changed, 312 insertions(+), 272 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> index f3129b912714..5ce6ba24fc72 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> @@ -614,8 +614,7 @@ static int init_user_pages(struct kgd_mem *mem, struct 
> mm_struct *mm,
>   amdgpu_bo_unreserve(bo);
>  
>  release_out:
> - if (ret)
> - release_pages(mem->user_pages, bo->tbo.ttm->num_pages);
> + amdgpu_ttm_tt_get_user_pages_done(bo->tbo.ttm);
>  free_out:
>   kvfree(mem->user_pages);
>   mem->user_pages = NULL;
> @@ -677,7 +676,6 @@ static int reserve_bo_and_vm(struct kgd_mem *mem,
>   ctx->kfd_bo.priority = 0;
>   ctx->kfd_bo.tv.bo = &bo->tbo;
>   ctx->kfd_bo.tv.shared = true;
> - ctx->kfd_bo.user_pages = NULL;
>   list_add(&ctx->kfd_bo.tv.head, &ctx->list);
>  
>   amdgpu_vm_get_pd_bo(vm, &ctx->list, &ctx->vm_pd[0]);
> @@ -741,7 +739,6 @@ static int reserve_bo_and_cond_vms(struct kgd_mem *mem,
>   ctx->kfd_bo.priority = 0;
>   ctx->kfd_bo.tv.bo = &bo->tbo;
>   ctx->kfd_bo.tv.shared = true;
> - ctx->kfd_bo.user_pages = NULL;
>   list_add(&ctx->kfd_bo.tv.head, &ctx->list);
>  
>   i = 0;
> @@ -1332,9 +1329,6 @@ int amdgpu_amdkfd_gpuvm_free_memory_of_gpu(
>   /* Free user pages if necessary */
>   if (mem->user_pages) {
>   pr_debug("%s: Freeing user_pages array\n", __func__);
> - if (mem->user_pages[0])
> - release_pages(mem->user_pages,
> - mem->bo->tbo.ttm->num_pages);
>   kvfree(mem->user_pages);
>   }
>  
> @@ -1761,8 +1755,6 @@ static int update_invalid_user_pages(struct 
> amdkfd_process_info *process_info,
>  __func__);
>   return -ENOMEM;
>   }
> - } else if (mem->user_pages[0]) {
> - release_pages(mem->user_pages, bo->tbo.ttm->num_pages);
>   }
>  
>   /* Get updated user pages */
> @@ -1778,12 +1770,6 @@ static int update_invalid_user_pages(struct 
> amdkfd_process_info *process_info,
>* stalled user mode queues.
>*/
>   }
> -
> - /* Mark the BO as valid unless it was invalidated
> -  * again concurrently
> -  */
> - if (atomic_cmpxchg(&mem->invalid, invalid, 0) != invalid)
> - return -EAGAIN;
>   }
>  
>   return 0;
> @@ -1876,14 +1862,10 @@ static int validate_invalid_user_pages(struct 
> amdkfd_process_info *process_info)
>   }
>  
>   /* Validate succeeded, now the BO owns the pages, free
> -  * our copy of the pointer array. Put this BO back on
> -  * the userptr_valid_list. If we need to revalidate
> -

Re: [PATCH 08/10] drm/amdgpu: add support for processing IH ring 1 & 2

[Kuehling, Felix]

Depending on the interrupt ring, the IRQ dispatch and processing
functions will run in interrupt context or in a worker thread.

Is there a way for the processing functions to find out which context
it's running in? That may influence decisions whether to process
interrupts in the same thread or schedule another worker.

Regards,
  Felix

On 2018-12-05 4:15 a.m., Christian König wrote:
> Previously we only added the ring buffer memory, now add the handling as
> well.
>
> Signed-off-by: Christian König 
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 33 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_irq.h |  4 ++-
>  2 files changed, 36 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c
> index b8e543e23166..8bfb3dab46f7 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c
> @@ -176,6 +176,36 @@ irqreturn_t amdgpu_irq_handler(int irq, void *arg)
>   return ret;
>  }
>  
> +/**
> + * amdgpu_irq_handle_ih1 - kick of processing for IH1
> + *
> + * @work: work structure in struct amdgpu_irq
> + *
> + * Kick of processing IH ring 1.
> + */
> +static void amdgpu_irq_handle_ih1(struct work_struct *work)
> +{
> + struct amdgpu_device *adev = container_of(work, struct amdgpu_device,
> +   irq.ih1_work);
> +
> + amdgpu_ih_process(adev, &adev->irq.ih1, amdgpu_irq_callback);
> +}
> +
> +/**
> + * amdgpu_irq_handle_ih2 - kick of processing for IH2
> + *
> + * @work: work structure in struct amdgpu_irq
> + *
> + * Kick of processing IH ring 2.
> + */
> +static void amdgpu_irq_handle_ih2(struct work_struct *work)
> +{
> + struct amdgpu_device *adev = container_of(work, struct amdgpu_device,
> +   irq.ih2_work);
> +
> + amdgpu_ih_process(adev, &adev->irq.ih2, amdgpu_irq_callback);
> +}
> +
>  /**
>   * amdgpu_msi_ok - check whether MSI functionality is enabled
>   *
> @@ -240,6 +270,9 @@ int amdgpu_irq_init(struct amdgpu_device *adev)
>   amdgpu_hotplug_work_func);
>   }
>  
> + INIT_WORK(&adev->irq.ih1_work, amdgpu_irq_handle_ih1);
> + INIT_WORK(&adev->irq.ih2_work, amdgpu_irq_handle_ih2);
> +
>   adev->irq.installed = true;
>   r = drm_irq_install(adev->ddev, adev->ddev->pdev->irq);
>   if (r) {
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.h 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.h
> index 7e06fa64321a..c27decfda494 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.h
> @@ -89,7 +89,9 @@ struct amdgpu_irq {
>  
>   /* interrupt rings */
>   struct amdgpu_ih_ring   ih, ih1, ih2;
> - const struct amdgpu_ih_funcs*ih_funcs;
> + const struct amdgpu_ih_funcs*ih_funcs;
> + struct work_struct  ih1_work, ih2_work;
> + struct amdgpu_irq_src   self_irq;
>  
>   /* gen irq stuff */
>   struct irq_domain   *domain; /* GPU irq controller domain */
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 09/10] drm/amdgpu: add support for self irq on Vega10

[Kuehling, Felix]

On 2018-12-05 4:15 a.m., Christian König wrote:
> This finally enables processing of ring 1 & 2.
>
> Signed-off-by: Christian König 
> ---
>  drivers/gpu/drm/amd/amdgpu/vega10_ih.c | 68 --
>  1 file changed, 63 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/vega10_ih.c 
> b/drivers/gpu/drm/amd/amdgpu/vega10_ih.c
> index 22638192d7dd..4a753e40a837 100644
> --- a/drivers/gpu/drm/amd/amdgpu/vega10_ih.c
> +++ b/drivers/gpu/drm/amd/amdgpu/vega10_ih.c
> @@ -272,7 +272,7 @@ static void vega10_ih_irq_disable(struct amdgpu_device 
> *adev)
>  static u32 vega10_ih_get_wptr(struct amdgpu_device *adev,
> struct amdgpu_ih_ring *ih)
>  {
> - u32 wptr, tmp;
> + u32 wptr, reg, tmp;
>  
>   wptr = le32_to_cpu(*ih->wptr_cpu);
>  
> @@ -288,9 +288,18 @@ static u32 vega10_ih_get_wptr(struct amdgpu_device *adev,
>wptr, ih->rptr, tmp);
>   ih->rptr = tmp;
>  
> - tmp = RREG32_NO_KIQ(SOC15_REG_OFFSET(OSSSYS, 0, mmIH_RB_CNTL));
> + if (ih == &adev->irq.ih)
> + reg = SOC15_REG_OFFSET(OSSSYS, 0, mmIH_RB_CNTL);
> + else if (ih == &adev->irq.ih1)
> + reg = SOC15_REG_OFFSET(OSSSYS, 0, mmIH_RB_CNTL);
> + else if (ih == &adev->irq.ih2)
> + reg = SOC15_REG_OFFSET(OSSSYS, 0, mmIH_RB_CNTL);

All three look the same to me. Did you mean to use mmIH_RB_CNTL_RING1/2
for ih1 and ih2 respectively?

Regards,
  Felix


> + else
> + BUG();
> +
> + tmp = RREG32_NO_KIQ(reg);
>   tmp = REG_SET_FIELD(tmp, IH_RB_CNTL, WPTR_OVERFLOW_CLEAR, 1);
> - WREG32_NO_KIQ(SOC15_REG_OFFSET(OSSSYS, 0, mmIH_RB_CNTL), tmp);
> + WREG32_NO_KIQ(reg, tmp);
>   }
>   return (wptr & ih->ptr_mask);
>  }
> @@ -352,9 +361,52 @@ static void vega10_ih_set_rptr(struct amdgpu_device 
> *adev,
>   /* XXX check if swapping is necessary on BE */
>   *ih->rptr_cpu = ih->rptr;
>   WDOORBELL32(ih->doorbell_index, ih->rptr);
> - } else {
> + } else if (ih == &adev->irq.ih) {
>   WREG32_SOC15(OSSSYS, 0, mmIH_RB_RPTR, ih->rptr);
> + } else if (ih == &adev->irq.ih1) {
> + WREG32_SOC15(OSSSYS, 0, mmIH_RB_RPTR_RING1, ih->rptr);
> + } else if (ih == &adev->irq.ih2) {
> + WREG32_SOC15(OSSSYS, 0, mmIH_RB_RPTR_RING2, ih->rptr);
> + }
> +}
> +
> +/**
> + * vega10_ih_self_irq - dispatch work for ring 1 and 2
> + *
> + * @adev: amdgpu_device pointer
> + * @source: irq source
> + * @entry: IV with WPTR update
> + *
> + * Update the WPTR from the IV and schedule work to handle the entries.
> + */
> +static int vega10_ih_self_irq(struct amdgpu_device *adev,
> +   struct amdgpu_irq_src *source,
> +   struct amdgpu_iv_entry *entry)
> +{
> + uint32_t wptr = cpu_to_le32(entry->src_data[0]);
> +
> + switch (entry->ring_id) {
> + case 1:
> + *adev->irq.ih1.wptr_cpu = wptr;
> + schedule_work(&adev->irq.ih1_work);
> + break;
> + case 2:
> + *adev->irq.ih2.wptr_cpu = wptr;
> + schedule_work(&adev->irq.ih2_work);
> + break;
> + default: break;
>   }
> + return 0;
> +}
> +
> +static const struct amdgpu_irq_src_funcs vega10_ih_self_irq_funcs = {
> + .process = vega10_ih_self_irq,
> +};
> +
> +static void vega10_ih_set_self_irq_funcs(struct amdgpu_device *adev)
> +{
> + adev->irq.self_irq.num_types = 0;
> + adev->irq.self_irq.funcs = &vega10_ih_self_irq_funcs;
>  }
>  
>  static int vega10_ih_early_init(void *handle)
> @@ -362,13 +414,19 @@ static int vega10_ih_early_init(void *handle)
>   struct amdgpu_device *adev = (struct amdgpu_device *)handle;
>  
>   vega10_ih_set_interrupt_funcs(adev);
> + vega10_ih_set_self_irq_funcs(adev);
>   return 0;
>  }
>  
>  static int vega10_ih_sw_init(void *handle)
>  {
> - int r;
>   struct amdgpu_device *adev = (struct amdgpu_device *)handle;
> + int r;
> +
> + r = amdgpu_irq_add_id(adev, SOC15_IH_CLIENTID_IH, 0,
> +   &adev->irq.self_irq);
> + if (r)
> + return r;
>  
>   r = amdgpu_ih_ring_init(adev, &adev->irq.ih, 256 * 1024, true);
>   if (r)
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 01/10] drm/amdgpu: send IVs to the KFD only after processing them v3

[Kuehling, Felix]

Patches 1-3 are Reviewed-by: Felix Kuehling 

I applied all 10 patches and tested them with kfdtest on Fiji and
Vega10. It seems to not break anything obvious.

I think I found a problem in patch 9 and have a question about patch 8
regarding the context in which interrupt processing functions would run.
I sent separate emails for both.

Regards,
  Felix

On 2018-12-05 4:15 a.m., Christian König wrote:
> This allows us to filter out VM faults in the GMC code.
>
> v2: don't filter out all faults
> v3: fix copy&paste typo, send all IV to the KFD, don't change message level
>
> Signed-off-by: Christian König 
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 38 +++--
>  1 file changed, 17 insertions(+), 21 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c
> index 6b6524f04ce0..79b6f456f2c5 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c
> @@ -149,9 +149,6 @@ static void amdgpu_irq_callback(struct amdgpu_device 
> *adev,
>   if (!amdgpu_ih_prescreen_iv(adev))
>   return;
>  
> - /* Before dispatching irq to IP blocks, send it to amdkfd */
> - amdgpu_amdkfd_interrupt(adev, (const void *) &ih->ring[ring_index]);
> -
>   entry.iv_entry = (const uint32_t *)&ih->ring[ring_index];
>   amdgpu_ih_decode_iv(adev, &entry);
>  
> @@ -371,39 +368,38 @@ void amdgpu_irq_dispatch(struct amdgpu_device *adev,
>   unsigned client_id = entry->client_id;
>   unsigned src_id = entry->src_id;
>   struct amdgpu_irq_src *src;
> + bool handled = false;
>   int r;
>  
>   trace_amdgpu_iv(entry);
>  
>   if (client_id >= AMDGPU_IRQ_CLIENTID_MAX) {
>   DRM_DEBUG("Invalid client_id in IV: %d\n", client_id);
> - return;
> - }
>  
> - if (src_id >= AMDGPU_MAX_IRQ_SRC_ID) {
> + } else  if (src_id >= AMDGPU_MAX_IRQ_SRC_ID) {
>   DRM_DEBUG("Invalid src_id in IV: %d\n", src_id);
> - return;
> - }
>  
> - if (adev->irq.virq[src_id]) {
> + } else if (adev->irq.virq[src_id]) {
>   generic_handle_irq(irq_find_mapping(adev->irq.domain, src_id));
> - } else {
> - if (!adev->irq.client[client_id].sources) {
> - DRM_DEBUG("Unregistered interrupt client_id: %d src_id: 
> %d\n",
> -   client_id, src_id);
> - return;
> - }
>  
> - src = adev->irq.client[client_id].sources[src_id];
> - if (!src) {
> - DRM_DEBUG("Unhandled interrupt src_id: %d\n", src_id);
> - return;
> - }
> + } else if (!adev->irq.client[client_id].sources) {
> + DRM_DEBUG("Unregistered interrupt client_id: %d src_id: %d\n",
> +   client_id, src_id);
>  
> + } else if ((src = adev->irq.client[client_id].sources[src_id])) {
>   r = src->funcs->process(adev, src, entry);
> - if (r)
> + if (r < 0)
>   DRM_ERROR("error processing interrupt (%d)\n", r);
> + else if (r)
> + handled = true;
> +
> + } else {
> + DRM_DEBUG("Unhandled interrupt src_id: %d\n", src_id);
>   }
> +
> + /* Send it to amdkfd as well if it isn't already handled */
> + if (!handled)
> + amdgpu_amdkfd_interrupt(adev, entry->iv_entry);
>  }
>  
>  /**
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 4/9] drm/amdgpu: Add a bitmask in amdgpu_ctx_mgr

[Kuehling, Felix]

On 2018-12-06 6:32 a.m., Rex Zhu wrote:
> used to manager the reserverd vm space.
>
> Signed-off-by: Rex Zhu 
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c | 8 ++--
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.h | 4 +++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 6 +-
>  3 files changed, 14 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
> index 8edf54b..8802ff2 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
> @@ -529,10 +529,14 @@ int amdgpu_ctx_wait_prev_fence(struct amdgpu_ctx *ctx,
>   return 0;
>  }
>  
> -void amdgpu_ctx_mgr_init(struct amdgpu_ctx_mgr *mgr)
> +int amdgpu_ctx_mgr_init(struct amdgpu_ctx_mgr *mgr)
>  {
>   mutex_init(&mgr->lock);
>   idr_init(&mgr->ctx_handles);
> + mgr->resv_vm_bitmap = kzalloc(DIV_ROUND_UP(AMDGPU_VM_MAX_NUM_CTX, 
> BITS_PER_BYTE), GFP_KERNEL);

The bitmap is in (unsigned long) units. So you should round to multiples
of (unsigned long), not multiples of byte. Something like this:

mgr->resv_vm_bitmap = kzalloc(DIV_ROUND_UP(AMDGPU_VM_MAX_NUM_CTX,
BITS_PER_BYTE * sizeof(unsigned long)) * sizeof(unsigned long), GFP_KERNEL);

Regards,
  Felix


> + if (unlikely(!mgr->resv_vm_bitmap))
> + return -ENOMEM;
> + return 0;
>  }
>  
>  void amdgpu_ctx_mgr_entity_flush(struct amdgpu_ctx_mgr *mgr)
> @@ -601,7 +605,7 @@ void amdgpu_ctx_mgr_fini(struct amdgpu_ctx_mgr *mgr)
>   if (kref_put(&ctx->refcount, amdgpu_ctx_fini) != 1)
>   DRM_ERROR("ctx %p is still alive\n", ctx);
>   }
> -
> + kfree(mgr->resv_vm_bitmap);
>   idr_destroy(&mgr->ctx_handles);
>   mutex_destroy(&mgr->lock);
>  }
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.h 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.h
> index b3b012c..94ac951 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.h
> @@ -38,6 +38,7 @@ struct amdgpu_ctx_entity {
>  struct amdgpu_ctx {
>   struct kref refcount;
>   struct amdgpu_device*adev;
> +
>   unsignedreset_counter;
>   unsignedreset_counter_query;
>   uint32_tvram_lost_counter;
> @@ -56,6 +57,7 @@ struct amdgpu_ctx_mgr {
>   struct mutexlock;
>   /* protected by lock */
>   struct idr  ctx_handles;
> + unsigned long   *resv_vm_bitmap;
>  };
>  
>  extern const unsigned int amdgpu_ctx_num_entities[AMDGPU_HW_IP_NUM];
> @@ -80,7 +82,7 @@ int amdgpu_ctx_ioctl(struct drm_device *dev, void *data,
>  int amdgpu_ctx_wait_prev_fence(struct amdgpu_ctx *ctx,
>  struct drm_sched_entity *entity);
>  
> -void amdgpu_ctx_mgr_init(struct amdgpu_ctx_mgr *mgr);
> +int amdgpu_ctx_mgr_init(struct amdgpu_ctx_mgr *mgr);
>  void amdgpu_ctx_mgr_entity_fini(struct amdgpu_ctx_mgr *mgr);
>  void amdgpu_ctx_mgr_entity_flush(struct amdgpu_ctx_mgr *mgr);
>  void amdgpu_ctx_mgr_fini(struct amdgpu_ctx_mgr *mgr);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
> index 52e4e90..338a091 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
> @@ -988,11 +988,15 @@ int amdgpu_driver_open_kms(struct drm_device *dev, 
> struct drm_file *file_priv)
>   mutex_init(&fpriv->bo_list_lock);
>   idr_init(&fpriv->bo_list_handles);
>  
> - amdgpu_ctx_mgr_init(&fpriv->ctx_mgr);
> + if (amdgpu_ctx_mgr_init(&fpriv->ctx_mgr))
> + goto error_ctx_mgr;
>  
>   file_priv->driver_priv = fpriv;
>   goto out_suspend;
>  
> +error_ctx_mgr:
> + idr_destroy(&fpriv->bo_list_handles);
> + mutex_destroy(&fpriv->bo_list_lock);
>  error_vm:
>   amdgpu_vm_fini(adev, &fpriv->vm);
>  
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 04/16 v2] drm/amd/display: Add tracing to dc

[Kuehling, Felix]

This change seems to be breaking the build for me. I'm getting errors like this:

  CC [M]  drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm.o
In file included from ../include/trace/events/tlb.h:9:0,
 from ../arch/x86/include/asm/mmu_context.h:10,
 from ../include/linux/mmu_context.h:5,
 from ../drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h:30,
 from ../drivers/gpu/drm/amd/amdgpu/amdgpu.h:85,
 from 
../drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm.c:34:
../include/linux/tracepoint.h:285:20: error: redefinition of 
â__tpstrtab_amdgpu_dc_rregâ
  static const char __tpstrtab_##name[] \
^
../include/linux/tracepoint.h:293:2: note: in expansion of macro 
âDEFINE_TRACE_FNâ
  DEFINE_TRACE_FN(name, NULL, NULL);
  ^
../include/trace/define_trace.h:28:2: note: in expansion of macro âDEFINE_TRACEâ
  DEFINE_TRACE(name)
  ^
../drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/./amdgpu_dm_trace.h:34:1: 
note: in expansion of macro âTRACE_EVENTâ
 TRACE_EVENT(amdgpu_dc_rreg,
 ^

I have a local change that adds #include  to amdgpu.h, which 
pulls in include/trace/events/tlb.h. That seems to create some kind of conflict 
with trace definitions. Any ideas how to fix this? It seems a bit fragile if 
adding some random include can break the build like this.

Thanks,
  Felix

-Original Message-
From: amd-gfx  On Behalf Of David Francis
Sent: Friday, November 30, 2018 9:57 AM
To: amd-gfx@lists.freedesktop.org
Cc: Francis, David 
Subject: [PATCH 04/16 v2] drm/amd/display: Add tracing to dc

[Why]
Tracing is a useful and cheap debug functionality

[How]
This creates a new trace system amdgpu_dm, currently with three trace events

amdgpu_dc_rreg and amdgpu_dc_wreg report the address and value of any dc 
register reads and writes

amdgpu_dc_performance requires at least one of those two to be enabled.  It 
counts the register reads and writes since the last entry

v2: Don't check for NULL before kfree

Signed-off-by: David Francis 
Reviewed-by: Harry Wentland 
Acked-by: Leo Li 
---
 .../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c |   3 +
 .../amd/display/amdgpu_dm/amdgpu_dm_trace.h   | 104 ++
 drivers/gpu/drm/amd/display/dc/core/dc.c  |  19 
 drivers/gpu/drm/amd/display/dc/dc_types.h |   8 ++
 .../amd/display/dc/dcn10/dcn10_cm_common.c|   4 +-
 drivers/gpu/drm/amd/display/dc/dm_services.h  |  12 +-
 6 files changed, 146 insertions(+), 4 deletions(-)  create mode 100644 
drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_trace.h

diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c 
b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
index 76b1aebdca0c..376927c8bcc6 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
@@ -23,6 +23,9 @@
  *
  */
 
+/* The caprices of the preprocessor require that this be declared right 
+here */ #define CREATE_TRACE_POINTS
+
 #include "dm_services_types.h"
 #include "dc.h"
 #include "dc/inc/core_types.h"
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_trace.h 
b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_trace.h
new file mode 100644
index ..d898981684d5
--- /dev/null
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_trace.h
@@ -0,0 +1,104 @@
+/*
+ * Copyright 2018 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person 
+obtaining a
+ * copy of this software and associated documentation files (the 
+"Software"),
+ * to deal in the Software without restriction, including without 
+limitation
+ * the rights to use, copy, modify, merge, publish, distribute, 
+sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom 
+the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be 
+included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 
+EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 
+MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT 
+SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, 
+DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 
+OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 
+OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Authors: AMD
+ *
+ */
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM amdgpu_dm
+
+#if !defined(_AMDGPU_DM_TRACE_H) || defined(TRACE_HEADER_MULTI_READ) 
+#define _AMDGPU_DM_TRACE_H_
+
+#include 
+
+TRACE_EVENT(amdgpu_dc_rreg,
+   TP_PROTO(unsigned long *read_count, uint32_t reg, uint32_t value),
+   TP_ARGS(read_count, reg, value),
+   TP_STRUCT__entry(
+   __field(uint32_t, reg)
+   __field(uint3

Re: [PATCH 04/16 v2] drm/amd/display: Add tracing to dc

[Kuehling, Felix]

On 2018-12-07 9:46 a.m., Wentland, Harry wrote:
> On 2018-12-07 9:41 a.m., Wentland, Harry wrote:
>> On 2018-12-07 12:40 a.m., Kuehling, Felix wrote:
>>> This change seems to be breaking the build for me. I'm getting errors like 
>>> this:
>>>
>>> CC [M]  drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm.o
>>> In file included from ../include/trace/events/tlb.h:9:0,
>>>from ../arch/x86/include/asm/mmu_context.h:10,
>>>from ../include/linux/mmu_context.h:5,
>>>from ../drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h:30,
>>>from ../drivers/gpu/drm/amd/amdgpu/amdgpu.h:85,
>>>from 
>>> ../drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm.c:34:
>>> ../include/linux/tracepoint.h:285:20: error: redefinition of 
>>> â__tpstrtab_amdgpu_dc_rregâ
>>> static const char __tpstrtab_##name[] \
>>>   ^
>>> ../include/linux/tracepoint.h:293:2: note: in expansion of macro 
>>> âDEFINE_TRACE_FNâ
>>> DEFINE_TRACE_FN(name, NULL, NULL);
>>> ^
>>> ../include/trace/define_trace.h:28:2: note: in expansion of macro 
>>> âDEFINE_TRACEâ
>>> DEFINE_TRACE(name)
>>> ^
>>> ../drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/./amdgpu_dm_trace.h:34:1:
>>>  note: in expansion of macro âTRACE_EVENTâ
>>>TRACE_EVENT(amdgpu_dc_rreg,
>>>^
>>>
>>> I have a local change that adds #include  to amdgpu.h, 
>>> which pulls in include/trace/events/tlb.h. That seems to create some kind 
>>> of conflict with trace definitions. Any ideas how to fix this? It seems a 
>>> bit fragile if adding some random include can break the build like this.
>>>
>> That's the trace subsystem for you. David and I are trying to understand 
>> what's going on. I think the problem is that both tlb.h and 
>> amdgpu_dm_trace.h define trace events and we now include both here.
>>
>> I think we'd want to include neither trace events from amdgpu.h to avoid 
>> this problem in the future, so we'll probably have to (a) clean up the dc.h 
>> include to only contain what amdgpu.h needs and (b) clean up amdgpu_amdkfd.h 
>> to only contain what amdgpu.h needs. At the end amdgpu.h doesn't care about 
>> the tracer. The problem seems that dc.h and amdgpu_amdkfd.h are the main 
>> includes for our respective drivers but are also wholesale included in 
>> amdgpu.h.
>>
> Apologies for the spam.
>
> Just noticed that amdgpu.h includes only amdgpu_dm.h which is clean. The 
> problem is that including amdgpu.h from amdgpu_dm.c now pulls in your trace 
> events (from tlb.h) which we don't expect and care about. I think we should 
> make sure amdgpu.h won't include anything that defines TRACE_EVENTs.

amdgpu_amdkfd.h defines a macro that uses use_mm and unuse_mm. Therefore
it needs to include mmu_context.h, which pulls in the conflicting trace
events. Maybe we can move that into a different header file that doesn't
get included in amdgpu.h. Or find another way to avoid including
amdgpu_amdkfd.h in amdgpu.h.

Thanks,
  Felix



>
> Harry
>
>> Harry
>>
>>> Thanks,
>>> Felix
>>>
>>> -Original Message-
>>> From: amd-gfx  On Behalf Of David 
>>> Francis
>>> Sent: Friday, November 30, 2018 9:57 AM
>>> To: amd-gfx@lists.freedesktop.org
>>> Cc: Francis, David 
>>> Subject: [PATCH 04/16 v2] drm/amd/display: Add tracing to dc
>>>
>>> [Why]
>>> Tracing is a useful and cheap debug functionality
>>>
>>> [How]
>>> This creates a new trace system amdgpu_dm, currently with three trace events
>>>
>>> amdgpu_dc_rreg and amdgpu_dc_wreg report the address and value of any dc 
>>> register reads and writes
>>>
>>> amdgpu_dc_performance requires at least one of those two to be enabled.  It 
>>> counts the register reads and writes since the last entry
>>>
>>> v2: Don't check for NULL before kfree
>>>
>>> Signed-off-by: David Francis 
>>> Reviewed-by: Harry Wentland 
>>> Acked-by: Leo Li 
>>> ---
>>>.../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c |   3 +
>>>.../amd/display/amdgpu_dm/amdgpu_dm_trace.h   | 104 ++
>>>drivers/gpu/drm/amd/display/dc/core/dc.c  |  19 
>>>drivers/gpu/drm/amd/display/dc/dc_types.h |   8 ++
>>>.../amd/d

Re: [PATCH 1/2] drm/amdgpu: add some additional vega10 pci ids

[Kuehling, Felix]

Can you add them amdkfd/kfd_device.c as well while you're at it.

Thanks,
  Felix

On 2018-12-07 4:03 p.m., Alex Deucher wrote:
> New vega ids.
>
> Signed-off-by: Alex Deucher 
> Cc: sta...@vger.kernel.org
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 6 ++
>  1 file changed, 6 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> index 90f474f98b6e..e32f6f43a46b 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> @@ -873,7 +873,13 @@ static const struct pci_device_id pciidlist[] = {
>   {0x1002, 0x6864, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA10},
>   {0x1002, 0x6867, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA10},
>   {0x1002, 0x6868, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA10},
> + {0x1002, 0x6869, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA10},
> + {0x1002, 0x686a, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA10},
> + {0x1002, 0x686b, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA10},
>   {0x1002, 0x686c, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA10},
> + {0x1002, 0x686d, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA10},
> + {0x1002, 0x686e, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA10},
> + {0x1002, 0x686f, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA10},
>   {0x1002, 0x687f, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA10},
>   /* Vega 12 */
>   {0x1002, 0x69A0, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA12},
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx