Re: drm/amdkfd: Relocate TBA/TMA to opposite side of VM hole (v2)
Hi Felix, Sorry for my late reply. I was busy this week. I just did some more tests using next-20240202 branch. Testing using blender 4.0.2, when only one HIP render task is running, there's no problem. However, when two tasks run together, software always crashes, but not crashes the whole system. Dmesg reports gpu reset in most cases, for example: [ 176.071823] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=32608, emitted seq=32610 [ 176.072000] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process blender pid 4256 thread blender:cs0 pid 4297 [ 176.072143] amdgpu :03:00.0: amdgpu: GPU reset begin! [ 176.073571] amdgpu :03:00.0: amdgpu: Guilty job already signaled, skipping HW reset [ 176.073593] amdgpu :03:00.0: amdgpu: GPU reset(4) succeeded! And in some rare cases, there would be a page fault report, see dmesg.log. Do you have any idea? Can I make it print more detailed diagnostic information? Best regards, Shengyu 在 2024/1/30 01:47, Felix Kuehling 写道: On 2024-01-29 10:24, Shengyu Qu wrote: Hello Felix, I think you are right. This problem has existed for years(just look at the issue creation time in my link), and is thought caused by OpenGL-ROCM interop(that's why I think this patch might help). It is very easy to trigger this problem in blender(method is also mentioned in the link). This doesn't help you, but it's unlikely that this has been the same issue for two years for everybody who chimed into this bug report. Different kernel versions, GPUs, user mode ROCm and Mesa versions etc. Case in point, it's possible that you're seeing an issue specific to RDNA3, which hasn't even been around for that long. Do you have any idea about this? Not without seeing a lot more diagnostic information. A full backtrace from your kernel log would be a good start. Regards, Felix Best regards, Shengyu 在 2024/1/29 22:51, Felix Kuehling 写道: On 2024-01-29 8:58, Shengyu Qu wrote: Hi, Seems rocm-opengl interop hang problem still exists[1]. Btw have you discovered into this problem? Best regards, Shengyu [1] https://projects.blender.org/blender/blender/issues/100353#issuecomment-599 Maybe you're having a different problem. Do you see this issue also without any version of the "Relocate TBA/TMA ..." patch? Regards, Felix 在 2024/1/27 03:15, Shengyu Qu 写道: Hello Felix, This patch seems working on my system, also it seems fixes the ROCM/OpenGL interop problem. Is this intended to happen or not? Maybe we need more users to test it. Besides, Tested-by: Shengyu Qu Best Regards, Shengyu 在 2024/1/26 06:27, Felix Kuehling 写道: The TBA and TMA, along with an unused IB allocation, reside at low addresses in the VM address space. A stray VM fault which hits these pages must be serviced by making their page table entries invalid. The scheduler depends upon these pages being resident and fails, preventing a debugger from inspecting the failure state. By relocating these pages above 47 bits in the VM address space they can only be reached when bits [63:48] are set to 1. This makes it much less likely for a misbehaving program to generate accesses to them. The current placement at VA (PAGE_SIZE*2) is readily hit by a NULL access with a small offset. v2: - Move it to the reserved space to avoid concflicts with Mesa - Add macros to make reserved space management easier Cc: Arunpravin Paneer Selvam Cc: Christian Koenig Signed-off-by: Jay Cornwall Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c | 7 ++--- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 12 ++-- drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 30 +++- 4 files changed, 30 insertions(+), 23 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c index 823d31f4a2a3..53d0a458d78e 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c @@ -28,9 +28,9 @@ uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) { - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; + uint64_t addr = AMDGPU_VA_RESERVED_CSA_START( + adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT); - addr -= AMDGPU_VA_RESERVED_CSA_SIZE; addr = amdgpu_gmc_sign_extend(addr); return addr; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c index 3d0d56087d41..9e769ef50f2e 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c @@ -45,11 +45,8 @@ */ static inline u64 amdgpu_seq64_get_va_base(struct amdgpu_device *adev) { - u64 addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; - - addr -= AMDGPU_VA_RESERVED_TOP;
Re: drm/amdkfd: Relocate TBA/TMA to opposite side of VM hole (v2)
Hi Alexander, 在 2024/2/6 1:12, Deucher, Alexander 写道: Are you only seeing the problem with this patch applied or in general? If you are seeing it in general, it likely related to a firmware issue that was recently fixed that will be resolved with an update CP firmware image. Driver side changes: https://gitlab.freedesktop.org/agd5f/linux/-/commit/0eb6c664b780dd1b4080e047ad51b100cd7840a3 https://gitlab.freedesktop.org/agd5f/linux/-/commit/40970e60070ed3d1390ec65e38e819f6d81b8f0c Alex This problem is not affected by this patch, so possible the firmware issue. Where can I get the newest firmware image? Or is it already pushed to linux-firmware repo? Best regards, Shengyu OpenPGP_0xE3520CC91929C8E7.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: drm/amdkfd: Relocate TBA/TMA to opposite side of VM hole (v2)
Hello Felix, This patch seems working on my system, also it seems fixes the ROCM/OpenGL interop problem. Is this intended to happen or not? Maybe we need more users to test it. Besides, Tested-by: Shengyu Qu Best Regards, Shengyu 在 2024/1/26 06:27, Felix Kuehling 写道: The TBA and TMA, along with an unused IB allocation, reside at low addresses in the VM address space. A stray VM fault which hits these pages must be serviced by making their page table entries invalid. The scheduler depends upon these pages being resident and fails, preventing a debugger from inspecting the failure state. By relocating these pages above 47 bits in the VM address space they can only be reached when bits [63:48] are set to 1. This makes it much less likely for a misbehaving program to generate accesses to them. The current placement at VA (PAGE_SIZE*2) is readily hit by a NULL access with a small offset. v2: - Move it to the reserved space to avoid concflicts with Mesa - Add macros to make reserved space management easier Cc: Arunpravin Paneer Selvam Cc: Christian Koenig Signed-off-by: Jay Cornwall Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c| 7 ++--- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 12 ++-- drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 30 +++- 4 files changed, 30 insertions(+), 23 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c index 823d31f4a2a3..53d0a458d78e 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c @@ -28,9 +28,9 @@ uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) { - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; + uint64_t addr = AMDGPU_VA_RESERVED_CSA_START( + adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT); - addr -= AMDGPU_VA_RESERVED_CSA_SIZE; addr = amdgpu_gmc_sign_extend(addr); return addr; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c index 3d0d56087d41..9e769ef50f2e 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c @@ -45,11 +45,8 @@ */ static inline u64 amdgpu_seq64_get_va_base(struct amdgpu_device *adev) { - u64 addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; - - addr -= AMDGPU_VA_RESERVED_TOP; - - return addr; + return AMDGPU_VA_RESERVED_SEQ64_START( + adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT); } /** diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h index 98a57192..f23b6153d310 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h @@ -135,11 +135,19 @@ struct amdgpu_mem_stats; #define AMDGPU_IS_MMHUB0(x) ((x) >= AMDGPU_MMHUB0_START && (x) < AMDGPU_MMHUB1_START) #define AMDGPU_IS_MMHUB1(x) ((x) >= AMDGPU_MMHUB1_START && (x) < AMDGPU_MAX_VMHUBS) -/* Reserve 2MB at top/bottom of address space for kernel use */ +/* Reserve space at top/bottom of address space for kernel use */ #define AMDGPU_VA_RESERVED_CSA_SIZE (2ULL << 20) +#define AMDGPU_VA_RESERVED_CSA_START(top) ((top) \ +- AMDGPU_VA_RESERVED_CSA_SIZE) #define AMDGPU_VA_RESERVED_SEQ64_SIZE (2ULL << 20) +#define AMDGPU_VA_RESERVED_SEQ64_START(top) (AMDGPU_VA_RESERVED_CSA_START(top) \ +- AMDGPU_VA_RESERVED_SEQ64_SIZE) +#define AMDGPU_VA_RESERVED_TRAP_SIZE (2ULL << 12) +#define AMDGPU_VA_RESERVED_TRAP_START(top) (AMDGPU_VA_RESERVED_SEQ64_START(top) \ +- AMDGPU_VA_RESERVED_TRAP_SIZE) #define AMDGPU_VA_RESERVED_BOTTOM (2ULL << 20) -#define AMDGPU_VA_RESERVED_TOP (AMDGPU_VA_RESERVED_SEQ64_SIZE + \ +#define AMDGPU_VA_RESERVED_TOP (AMDGPU_VA_RESERVED_TRAP_SIZE + \ +AMDGPU_VA_RESERVED_SEQ64_SIZE + \ AMDGPU_VA_RESERVED_CSA_SIZE) /* See vm_update_mode */ diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c index 6604a3f99c5e..f899cce25b2a 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c @@ -36,6 +36,7 @@ #include #include #include +#include "amdgpu_vm.h" /* * The primary memory I/O features being added for revisions of gfxip @@ -326,10 +327,16 @@ static void kfd_init_apertures_vi(struct kfd_process_device *pdd, uint8_t id) * with small reserved space for kernel. *
Re: drm/amdkfd: Relocate TBA/TMA to opposite side of VM hole (v2)
Hello Felix, I think you are right. This problem has existed for years(just look at the issue creation time in my link), and is thought caused by OpenGL-ROCM interop(that's why I think this patch might help). It is very easy to trigger this problem in blender(method is also mentioned in the link). Do you have any idea about this? Best regards, Shengyu 在 2024/1/29 22:51, Felix Kuehling 写道: On 2024-01-29 8:58, Shengyu Qu wrote: Hi, Seems rocm-opengl interop hang problem still exists[1]. Btw have you discovered into this problem? Best regards, Shengyu [1] https://projects.blender.org/blender/blender/issues/100353#issuecomment-599 Maybe you're having a different problem. Do you see this issue also without any version of the "Relocate TBA/TMA ..." patch? Regards, Felix 在 2024/1/27 03:15, Shengyu Qu 写道: Hello Felix, This patch seems working on my system, also it seems fixes the ROCM/OpenGL interop problem. Is this intended to happen or not? Maybe we need more users to test it. Besides, Tested-by: Shengyu Qu Best Regards, Shengyu 在 2024/1/26 06:27, Felix Kuehling 写道: The TBA and TMA, along with an unused IB allocation, reside at low addresses in the VM address space. A stray VM fault which hits these pages must be serviced by making their page table entries invalid. The scheduler depends upon these pages being resident and fails, preventing a debugger from inspecting the failure state. By relocating these pages above 47 bits in the VM address space they can only be reached when bits [63:48] are set to 1. This makes it much less likely for a misbehaving program to generate accesses to them. The current placement at VA (PAGE_SIZE*2) is readily hit by a NULL access with a small offset. v2: - Move it to the reserved space to avoid concflicts with Mesa - Add macros to make reserved space management easier Cc: Arunpravin Paneer Selvam Cc: Christian Koenig Signed-off-by: Jay Cornwall Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c | 7 ++--- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 12 ++-- drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 30 +++- 4 files changed, 30 insertions(+), 23 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c index 823d31f4a2a3..53d0a458d78e 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c @@ -28,9 +28,9 @@ uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) { - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; + uint64_t addr = AMDGPU_VA_RESERVED_CSA_START( + adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT); - addr -= AMDGPU_VA_RESERVED_CSA_SIZE; addr = amdgpu_gmc_sign_extend(addr); return addr; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c index 3d0d56087d41..9e769ef50f2e 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c @@ -45,11 +45,8 @@ */ static inline u64 amdgpu_seq64_get_va_base(struct amdgpu_device *adev) { - u64 addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; - - addr -= AMDGPU_VA_RESERVED_TOP; - - return addr; + return AMDGPU_VA_RESERVED_SEQ64_START( + adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT); } /** diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h index 98a57192..f23b6153d310 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h @@ -135,11 +135,19 @@ struct amdgpu_mem_stats; #define AMDGPU_IS_MMHUB0(x) ((x) >= AMDGPU_MMHUB0_START && (x) < AMDGPU_MMHUB1_START) #define AMDGPU_IS_MMHUB1(x) ((x) >= AMDGPU_MMHUB1_START && (x) < AMDGPU_MAX_VMHUBS) -/* Reserve 2MB at top/bottom of address space for kernel use */ +/* Reserve space at top/bottom of address space for kernel use */ #define AMDGPU_VA_RESERVED_CSA_SIZE (2ULL << 20) +#define AMDGPU_VA_RESERVED_CSA_START(top) ((top) \ + - AMDGPU_VA_RESERVED_CSA_SIZE) #define AMDGPU_VA_RESERVED_SEQ64_SIZE (2ULL << 20) +#define AMDGPU_VA_RESERVED_SEQ64_START(top) (AMDGPU_VA_RESERVED_CSA_START(top) \ + - AMDGPU_VA_RESERVED_SEQ64_SIZE) +#define AMDGPU_VA_RESERVED_TRAP_SIZE (2ULL << 12) +#define AMDGPU_VA_RESERVED_TRAP_START(top) (AMDGPU_VA_RESERVED_SEQ64_START(top) \ + - AMDGPU_VA_RESERVED_TRAP_SIZE) #define AMDGPU_VA_RESERVED_BOTTOM (2ULL << 20) -#define AMDGPU_VA_RESERVED_TOP (AMDGPU_VA_RESERVED_SEQ64_SIZE + \ +#define AMDGPU_VA_RESERVED_TOP (AMDGPU_VA_RESERVED_TRAP_SIZE + \ + AMDGPU_VA_RESERVED_SEQ64_SIZE + \
Re: drm/amdkfd: Relocate TBA/TMA to opposite side of VM hole (v2)
Hi, Seems rocm-opengl interop hang problem still exists[1]. Btw have you discovered into this problem? Best regards, Shengyu [1] https://projects.blender.org/blender/blender/issues/100353#issuecomment-599 在 2024/1/27 03:15, Shengyu Qu 写道: Hello Felix, This patch seems working on my system, also it seems fixes the ROCM/OpenGL interop problem. Is this intended to happen or not? Maybe we need more users to test it. Besides, Tested-by: Shengyu Qu Best Regards, Shengyu 在 2024/1/26 06:27, Felix Kuehling 写道: The TBA and TMA, along with an unused IB allocation, reside at low addresses in the VM address space. A stray VM fault which hits these pages must be serviced by making their page table entries invalid. The scheduler depends upon these pages being resident and fails, preventing a debugger from inspecting the failure state. By relocating these pages above 47 bits in the VM address space they can only be reached when bits [63:48] are set to 1. This makes it much less likely for a misbehaving program to generate accesses to them. The current placement at VA (PAGE_SIZE*2) is readily hit by a NULL access with a small offset. v2: - Move it to the reserved space to avoid concflicts with Mesa - Add macros to make reserved space management easier Cc: Arunpravin Paneer Selvam Cc: Christian Koenig Signed-off-by: Jay Cornwall Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c | 7 ++--- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 12 ++-- drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 30 +++- 4 files changed, 30 insertions(+), 23 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c index 823d31f4a2a3..53d0a458d78e 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c @@ -28,9 +28,9 @@ uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) { - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; + uint64_t addr = AMDGPU_VA_RESERVED_CSA_START( + adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT); - addr -= AMDGPU_VA_RESERVED_CSA_SIZE; addr = amdgpu_gmc_sign_extend(addr); return addr; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c index 3d0d56087d41..9e769ef50f2e 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c @@ -45,11 +45,8 @@ */ static inline u64 amdgpu_seq64_get_va_base(struct amdgpu_device *adev) { - u64 addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; - - addr -= AMDGPU_VA_RESERVED_TOP; - - return addr; + return AMDGPU_VA_RESERVED_SEQ64_START( + adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT); } /** diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h index 98a57192..f23b6153d310 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h @@ -135,11 +135,19 @@ struct amdgpu_mem_stats; #define AMDGPU_IS_MMHUB0(x) ((x) >= AMDGPU_MMHUB0_START && (x) < AMDGPU_MMHUB1_START) #define AMDGPU_IS_MMHUB1(x) ((x) >= AMDGPU_MMHUB1_START && (x) < AMDGPU_MAX_VMHUBS) -/* Reserve 2MB at top/bottom of address space for kernel use */ +/* Reserve space at top/bottom of address space for kernel use */ #define AMDGPU_VA_RESERVED_CSA_SIZE (2ULL << 20) +#define AMDGPU_VA_RESERVED_CSA_START(top) ((top) \ + - AMDGPU_VA_RESERVED_CSA_SIZE) #define AMDGPU_VA_RESERVED_SEQ64_SIZE (2ULL << 20) +#define AMDGPU_VA_RESERVED_SEQ64_START(top) (AMDGPU_VA_RESERVED_CSA_START(top) \ + - AMDGPU_VA_RESERVED_SEQ64_SIZE) +#define AMDGPU_VA_RESERVED_TRAP_SIZE (2ULL << 12) +#define AMDGPU_VA_RESERVED_TRAP_START(top) (AMDGPU_VA_RESERVED_SEQ64_START(top) \ + - AMDGPU_VA_RESERVED_TRAP_SIZE) #define AMDGPU_VA_RESERVED_BOTTOM (2ULL << 20) -#define AMDGPU_VA_RESERVED_TOP (AMDGPU_VA_RESERVED_SEQ64_SIZE + \ +#define AMDGPU_VA_RESERVED_TOP (AMDGPU_VA_RESERVED_TRAP_SIZE + \ + AMDGPU_VA_RESERVED_SEQ64_SIZE + \ AMDGPU_VA_RESERVED_CSA_SIZE) /* See vm_update_mode */ diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c index 6604a3f99c5e..f899cce25b2a 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c @@ -36,6 +36,7 @@ #include #include #include +#include "amdgpu_vm.h" /* * The primary memory I/O features being added for revisions of gfxip @@ -326,10 +327,16 @@ static void kfd_init_apertures_vi(struct kfd_process_device *pdd, uint8_t id) *
Re: drm/amdkfd: Relocate TBA/TMA to opposite side of VM hole (v2)
Hi Felix, Thanks for reply. I'll record a backtrace when I'm free. Besides, here is a dmesg log from someone else in the issue discussion about this problem: https://projects.blender.org/attachments/ea7b7db5-ac16-479d-935b-9e1da33cd6f0 Tested using next-20240129 with this patch applied, and setup is Plasma 6.0 RC1(Wayland) + RX 6600 XT. Best regards, Shengyu 在 2024/1/30 1:47, Felix Kuehling 写道: > On 2024-01-29 10:24, Shengyu Qu wrote: >> Hello Felix, >> I think you are right. This problem has existed for years(just look at >> the >> issue creation time in my link), and is thought caused by OpenGL-ROCMTe >> interop(that's why I think this patch might help). It is very easy to >> trigger this problem in blender(method is also mentioned in the link). > > This doesn't help you, but it's unlikely that this has been the same > issue for two years for everybody who chimed into this bug report. > Different kernel versions, GPUs, user mode ROCm and Mesa versions etc. > > Case in point, it's possible that you're seeing an issue specific to > RDNA3, which hasn't even been around for that long. > > >> Do >> you have any idea about this? > > Not without seeing a lot more diagnostic information. A full backtrace > from your kernel log would be a good start. > > Regards, > Felix > > >> Best regards, >> Shengyu >> 在 2024/1/29 22:51, Felix Kuehling 写道: >>> On 2024-01-29 8:58, Shengyu Qu wrote: >>>> Hi, >>>> Seems rocm-opengl interop hang problem still exists[1]. Btw have you >>>> discovered into this problem? >>>> Best regards, >>>> Shengyu >>>> [1] >>>> https://projects.blender.org/blender/blender/issues/100353#issuecomment-599 >>> >>> Maybe you're having a different problem. Do you see this issue also >>> without any version of the "Relocate TBA/TMA ..." patch? >>> >>> Regards, >>> Felix >>> >>> >>>> >>>> 在 2024/1/27 03:15, Shengyu Qu 写道: >>>>> Hello Felix, >>>>> This patch seems working on my system, also it seems fixes the >>>>> ROCM/OpenGL >>>>> interop problem. >>>>> Is this intended to happen or not? Maybe we need more users to test >>>>> it. >>>>> Besides, >>>>> Tested-by: Shengyu Qu >>>>> Best Regards, >>>>> Shengyu >>>>> >>>>> 在 2024/1/26 06:27, Felix Kuehling 写道: >>>>>> The TBA and TMA, along with an unused IB allocation, reside at low >>>>>> addresses in the VM address space. A stray VM fault which hits these >>>>>> pages must be serviced by making their page table entries invalid. >>>>>> The scheduler depends upon these pages being resident and fails, >>>>>> preventing a debugger from inspecting the failure state. >>>>>> >>>>>> By relocating these pages above 47 bits in the VM address space they >>>>>> can only be reached when bits [63:48] are set to 1. This makes it >>>>>> much >>>>>> less likely for a misbehaving program to generate accesses to them. >>>>>> The current placement at VA (PAGE_SIZE*2) is readily hit by a NULL >>>>>> access with a small offset. >>>>>> >>>>>> v2: >>>>>> - Move it to the reserved space to avoid concflicts with Mesa >>>>>> - Add macros to make reserved space management easier >>>>>> >>>>>> Cc: Arunpravin Paneer Selvam >>>>>> Cc: Christian Koenig >>>>>> Signed-off-by: Jay Cornwall >>>>>> Signed-off-by: Felix Kuehling >>>>>> --- >>>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 4 +-- >>>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c | 7 ++--- >>>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 12 ++-- >>>>>> drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 30 >>>>>> +++- >>>>>> 4 files changed, 30 insertions(+), 23 deletions(-) >>>>>> >>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >>>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >>>>>> index 823d31f4a2a3..53d0a458d78e 100644 >>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >>>>>> +++ b/drivers/gp
Re: [PATCH V8 24/43] drm/amd/display: Skip color pipeline initialization for cursor plane
Btw what's your opinion about this, Xaver? 在 2025/4/1 17:56, Michel Dänzer 写道: On 2025-03-31 19:42, Alex Hung wrote: On 3/31/25 11:04, Shengyu Qu wrote: Or we can add some kind of "linked with" info to plane's COLOR_PIPELINE property, to let userspace know that cursor plane and background plane share the same colorop config. So that userspace could do extra conversion on cursor image data to avoid display wrong cursor color. That's over-complicate and makes little sense for both device drivers and userspace applications. If any planes share same colorop config, a device driver exposes the same color pipeline with the same colorops. If a plane does not support color pipeline or a driver doesn't want to support it, there is no color pipeline and no color objects. I suspect using the cursor plane is generally higher priority for Wayland compositors than using overlay planes, because the former is critical for a responsive user experience. This requires that the amdgpu DC driver backs the cursor plane with a dedicated HW plane though (as it's already doing in some cases), to either fully support color pipelines for the cursor plane, or at least provide proper "no color pipeline" behaviour for it. Letting the effective behaviour be determined by the other planes which happen to be behind the cursor plane isn't usable for Wayland compositors. Current behavior is just disable colorop for both background plane and cursor plane. I'm not sure how much planes does the hardware support, but if there are too less planes to use, maybe we still need to make use of the cursor background plane in the compositor. And then how to deal with colorop of cursor and background plane while saving as much power as possible becomes a problem. OpenPGP_0xE3520CC91929C8E7.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: [PATCH V8 24/43] drm/amd/display: Skip color pipeline initialization for cursor plane
Thanks, I mistook about the MPO document. Maybe we should also disable colorop on the background plane of the cursor plane? So that compositors would do sw color convertion on both cursor plane and background plane, which should keep cursor display correctly. 在 2025/4/1 0:34, Alex Hung 写道: On 3/31/25 10:31, Shengyu Qu wrote: Sorry for vague expression. I mean that I think we shouldn't register DRM_PLANE_TYPE_CURSOR in the driver, as we don't have actual hardware support. This is not true. AMD has hardware cursor support. 在 2025/4/1 0:26, Alex Hung 写道: On 3/31/25 10:12, Shengyu Qu wrote: So currently we have to hope the compositor won't use DRM_PLANE_TYPE_CURSOR planes at all Why do we still register DRM_PLANE_TYPE_CURSOR in the driver? I am not sure what your question is. A compositor can choose or skip any hardware features, but this discussion is out of the scope. 在 2025/4/1 0:06, Alex Hung 写道: On 3/31/25 09:43, Shengyu Qu wrote: Hi, Thanks for reply. So currently we have to apply color conversion on the background plane of the cursor to do some color space conversion. What would happen if cursor and background plane needs different conversion config? Or we just give the cursor a dedicated plane? This scenario is not supported on AMD hardware, but software cursors on other plane types won't be affected. Best regards, Shengyu 在 2025/3/31 22:28, Alex Hung 写道: On 3/30/25 06:59, Shengyu Qu wrote: Hi, Do we really need to disable cursor plane color pipeline support? I don't think we need to disable that if it is supported, since there might be some user-defined colored cursor icon. This patch applies to AMD hardware only: https:// elixir.bootlin.com/ linux/v6.13/source/Documentation/gpu/amdgpu/ display/mpo- overview.rst#L101 Best regards, Shengyu For some unknown reason, seems my mail is not shown in the mail list archive, so I resent it. 在 2025/3/27 7:47, Alex Hung 写道: cursor plane does not need to have color pipeline. Signed-off-by: Alex Hung --- v7: - Add a commit messages drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/ amdgpu_dm_plane.c b/drivers/gpu/drm/amd/display/amdgpu_dm/ amdgpu_dm_plane.c index 9632b8b73e7e..b5b9b0b5da63 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c @@ -1792,6 +1792,9 @@ dm_plane_init_colorops(struct drm_plane *plane) struct drm_prop_enum_list pipelines[MAX_COLOR_PIPELINES]; int len = 0; + if (plane->type == DRM_PLANE_TYPE_CURSOR) + return 0; + /* Create COLOR_PIPELINE property and attach */ drm_plane_create_color_pipeline_property(plane, pipelines, len); OpenPGP_0xE3520CC91929C8E7.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: [PATCH V8 24/43] drm/amd/display: Skip color pipeline initialization for cursor plane
Hi, Do we really need to disable cursor plane color pipeline support? I don't think we need to disable that if it is supported, since there might be some user-defined colored cursor icon. Best regards, Shengyu For some unknown reason, seems my mail is not shown in the mail list archive, so I resent it. 在 2025/3/27 7:47, Alex Hung 写道: cursor plane does not need to have color pipeline. Signed-off-by: Alex Hung --- v7: - Add a commit messages drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c index 9632b8b73e7e..b5b9b0b5da63 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c @@ -1792,6 +1792,9 @@ dm_plane_init_colorops(struct drm_plane *plane) struct drm_prop_enum_list pipelines[MAX_COLOR_PIPELINES]; int len = 0; + if (plane->type == DRM_PLANE_TYPE_CURSOR) + return 0; + /* Create COLOR_PIPELINE property and attach */ drm_plane_create_color_pipeline_property(plane, pipelines, len); OpenPGP_0xE3520CC91929C8E7.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: [PATCH V8 24/43] drm/amd/display: Skip color pipeline initialization for cursor plane
Hi, Do we really need to disable cursor plane color pipeline support? I don't think we need to disable that if it is supported, since there might be some user-defined colored cursor icon. Best regards, Shengyu 在 2025/3/27 7:47, Alex Hung 写道: cursor plane does not need to have color pipeline. Signed-off-by: Alex Hung --- v7: - Add a commit messages drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c index 9632b8b73e7e..b5b9b0b5da63 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c @@ -1792,6 +1792,9 @@ dm_plane_init_colorops(struct drm_plane *plane) struct drm_prop_enum_list pipelines[MAX_COLOR_PIPELINES]; int len = 0; + if (plane->type == DRM_PLANE_TYPE_CURSOR) + return 0; + /* Create COLOR_PIPELINE property and attach */ drm_plane_create_color_pipeline_property(plane, pipelines, len); OpenPGP_0xE3520CC91929C8E7.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: [PATCH V8 24/43] drm/amd/display: Skip color pipeline initialization for cursor plane
在 2025/4/1 22:11, Michel Dänzer 写道: On 2025-04-01 14:32, Shengyu Qu wrote: 在 2025/4/1 17:56, Michel Dänzer 写道: On 2025-03-31 19:42, Alex Hung wrote: On 3/31/25 11:04, Shengyu Qu wrote: Or we can add some kind of "linked with" info to plane's COLOR_PIPELINE property, to let userspace know that cursor plane and background plane share the same colorop config. So that userspace could do extra conversion on cursor image data to avoid display wrong cursor color. That's over-complicate and makes little sense for both device drivers and userspace applications. If any planes share same colorop config, a device driver exposes the same color pipeline with the same colorops. If a plane does not support color pipeline or a driver doesn't want to support it, there is no color pipeline and no color objects. I suspect using the cursor plane is generally higher priority for Wayland compositors than using overlay planes, because the former is critical for a responsive user experience. This requires that the amdgpu DC driver backs the cursor plane with a dedicated HW plane though (as it's already doing in some cases), to either fully support color pipelines for the cursor plane, or at least provide proper "no color pipeline" behaviour for it. Letting the effective behaviour be determined by the other planes which happen to be behind the cursor plane isn't usable for Wayland compositors. Current behavior is just disable colorop for both background plane and cursor plane. Are you saying the color pipeline is implicitly disabled for any KMS planes which happen to be overlapped by the cursor plane? Forget about my previous mail. Background plane means the parent plane of cursor plane, which is described here: https://docs.kernel.org/next/gpu/amdgpu/display/mpo-overview.html Not mean the plane that is overlapped by cursor plane, since only one plane is affected by cursor plane. That sounds like a no go. I'm not sure how much planes does the hardware support, but if there are too less planes to use, maybe we still need to make use of the cursor background plane in the compositor. If the HW has too few planes to allow both the cursor & overlay planes to work correctly (regardless of their dimensions), the driver should not allow enabling both kinds of planes at the same time. OpenPGP_0xE3520CC91929C8E7.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: [PATCH V8 24/43] drm/amd/display: Skip color pipeline initialization for cursor plane
在 2025/4/1 22:11, Michel Dänzer 写道: On 2025-04-01 14:32, Shengyu Qu wrote: 在 2025/4/1 17:56, Michel Dänzer 写道: On 2025-03-31 19:42, Alex Hung wrote: On 3/31/25 11:04, Shengyu Qu wrote: Or we can add some kind of "linked with" info to plane's COLOR_PIPELINE property, to let userspace know that cursor plane and background plane share the same colorop config. So that userspace could do extra conversion on cursor image data to avoid display wrong cursor color. That's over-complicate and makes little sense for both device drivers and userspace applications. If any planes share same colorop config, a device driver exposes the same color pipeline with the same colorops. If a plane does not support color pipeline or a driver doesn't want to support it, there is no color pipeline and no color objects. I suspect using the cursor plane is generally higher priority for Wayland compositors than using overlay planes, because the former is critical for a responsive user experience. This requires that the amdgpu DC driver backs the cursor plane with a dedicated HW plane though (as it's already doing in some cases), to either fully support color pipelines for the cursor plane, or at least provide proper "no color pipeline" behaviour for it. Letting the effective behaviour be determined by the other planes which happen to be behind the cursor plane isn't usable for Wayland compositors. Current behavior is just disable colorop for both background plane and cursor plane. Are you saying the color pipeline is implicitly disabled for any KMS planes which happen to be overlapped by the cursor plane? According to this mail, I think so(unless I mistook about the meaning again): https://lists.freedesktop.org/archives/amd-gfx/2025-April/122257.html That sounds like a no go. I'm not sure how much planes does the hardware support, but if there are too less planes to use, maybe we still need to make use of the cursor background plane in the compositor. If the HW has too few planes to allow both the cursor & overlay planes to work correctly (regardless of their dimensions), the driver should not allow enabling both kinds of planes at the same time. OpenPGP_0xE3520CC91929C8E7.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: [PATCH V8 11/43] drm/colorop: Introduce DRM_CLIENT_CAP_PLANE_COLOR_PIPELINE
在 2025/3/27 7:46, Alex Hung 写道: From: Harry Wentland With the introduction of the pre-blending color pipeline we can no longer have color operations that don't have a clear position in the color pipeline. We deprecate all existing plane properties. For upstream drivers those are: - COLOR_ENCODING - COLOR_RANGE Drivers are expected to ignore these properties when programming the HW. DRM clients that don't register with don't? Seems conflict with change note below. DRM_CLIENT_CAP_PLANE_COLOR_PIPELINE will not be allowed to set the COLOR_ENCODING and COLOR_RANGE properties. Setting of the COLOR_PIPELINE plane property or drm_colorop properties is only allowed for userspace that sets this client cap. Signed-off-by: Alex Hung Signed-off-by: Harry Wentland --- v8: - Disallow setting of COLOR_RANGE and COLOR_ENCODING when DRM_CLIENT_CAP_PLANE_COLOR_PIPELINE is set v5: - Fix kernel docs v4: - Don't block setting of COLOR_RANGE and COLOR_ENCODING when client cap is set drivers/gpu/drm/drm_atomic_uapi.c | 23 ++- drivers/gpu/drm/drm_ioctl.c | 7 +++ include/drm/drm_file.h| 7 +++ include/uapi/drm/drm.h| 15 +++ 4 files changed, 51 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/drm_atomic_uapi.c b/drivers/gpu/drm/drm_atomic_uapi.c index 5738b1c18755..e0b4b122ef6b 100644 --- a/drivers/gpu/drm/drm_atomic_uapi.c +++ b/drivers/gpu/drm/drm_atomic_uapi.c @@ -567,10 +567,26 @@ static int drm_atomic_plane_set_property(struct drm_plane *plane, } else if (property == plane->zpos_property) { state->zpos = val; } else if (property == plane->color_encoding_property) { + if (file_priv->plane_color_pipeline) { + drm_dbg_atomic(dev, + "Setting COLOR_ENCODING plane property not permitted with DRM_CLIENT_CAP_PLANE_COLOR_PIPELINE client cap\n"); + return -EINVAL; + } state->color_encoding = val; } else if (property == plane->color_range_property) { + if (file_priv->plane_color_pipeline) { + drm_dbg_atomic(dev, + "Setting COLOR_RANGE plane property not permitted with DRM_CLIENT_CAP_PLANE_COLOR_PIPELINE client cap\n"); + return -EINVAL; + } state->color_range = val; } else if (property == plane->color_pipeline_property) { + if (!file_priv->plane_color_pipeline) { + drm_dbg_atomic(dev, + "Setting COLOR_PIPELINE plane property not permitted unless DRM_CLIENT_CAP_PLANE_COLOR_PIPELINE is set\n"); + return -EINVAL; + } + /* find DRM colorop object */ struct drm_colorop *colorop = NULL; @@ -1197,6 +1213,12 @@ int drm_atomic_set_property(struct drm_atomic_state *state, break; } case DRM_MODE_OBJECT_COLOROP: { + if (!file_priv->plane_color_pipeline) { + drm_dbg_atomic(prop->dev, + "[OBJECT:%d] is a colorop but DRM_CLIENT_CAP_PLANE_COLOR_PIPELINE not set\n", + obj->id); + ret = -EINVAL; + } struct drm_colorop *colorop = obj_to_colorop(obj); struct drm_colorop_state *colorop_state; @@ -1209,7 +1231,6 @@ int drm_atomic_set_property(struct drm_atomic_state *state, ret = drm_atomic_colorop_set_property(colorop, colorop_state, file_priv, prop, prop_value); - break; } default: diff --git a/drivers/gpu/drm/drm_ioctl.c b/drivers/gpu/drm/drm_ioctl.c index f593dc569d31..5c89c586da7c 100644 --- a/drivers/gpu/drm/drm_ioctl.c +++ b/drivers/gpu/drm/drm_ioctl.c @@ -373,6 +373,13 @@ drm_setclientcap(struct drm_device *dev, void *data, struct drm_file *file_priv) return -EINVAL; file_priv->supports_virtualized_cursor_plane = req->value; break; + case DRM_CLIENT_CAP_PLANE_COLOR_PIPELINE: + if (!file_priv->atomic) + return -EINVAL; + if (req->value > 1) + return -EINVAL; + file_priv->plane_color_pipeline = req->value; + break; default: return -EINVAL; } diff --git a/include/drm/drm_file.h b/include/drm/drm_file.h index 94d365b22505..86929ca667aa 100644 --- a/include/drm/drm_file.h +++ b/include/drm/drm_file.h @@ -206,6 +206,13 @@ struct drm_file { */ bool writeback_connectors; + /** +* @plane_color_pipeline: +* +* True if client understands plane
Re: [PATCH V8 24/43] drm/amd/display: Skip color pipeline initialization for cursor plane
Sorry for vague expression. I mean that I think we shouldn't register DRM_PLANE_TYPE_CURSOR in the driver, as we don't have actual hardware support. 在 2025/4/1 0:26, Alex Hung 写道: On 3/31/25 10:12, Shengyu Qu wrote: So currently we have to hope the compositor won't use DRM_PLANE_TYPE_CURSOR planes at all Why do we still register DRM_PLANE_TYPE_CURSOR in the driver? I am not sure what your question is. A compositor can choose or skip any hardware features, but this discussion is out of the scope. 在 2025/4/1 0:06, Alex Hung 写道: On 3/31/25 09:43, Shengyu Qu wrote: Hi, Thanks for reply. So currently we have to apply color conversion on the background plane of the cursor to do some color space conversion. What would happen if cursor and background plane needs different conversion config? Or we just give the cursor a dedicated plane? This scenario is not supported on AMD hardware, but software cursors on other plane types won't be affected. Best regards, Shengyu 在 2025/3/31 22:28, Alex Hung 写道: On 3/30/25 06:59, Shengyu Qu wrote: Hi, Do we really need to disable cursor plane color pipeline support? I don't think we need to disable that if it is supported, since there might be some user-defined colored cursor icon. This patch applies to AMD hardware only: https:// elixir.bootlin.com/ linux/v6.13/source/Documentation/gpu/amdgpu/ display/mpo- overview.rst#L101 Best regards, Shengyu For some unknown reason, seems my mail is not shown in the mail list archive, so I resent it. 在 2025/3/27 7:47, Alex Hung 写道: cursor plane does not need to have color pipeline. Signed-off-by: Alex Hung --- v7: - Add a commit messages drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/ amdgpu_dm_plane.c b/drivers/gpu/drm/amd/display/amdgpu_dm/ amdgpu_dm_plane.c index 9632b8b73e7e..b5b9b0b5da63 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c @@ -1792,6 +1792,9 @@ dm_plane_init_colorops(struct drm_plane *plane) struct drm_prop_enum_list pipelines[MAX_COLOR_PIPELINES]; int len = 0; + if (plane->type == DRM_PLANE_TYPE_CURSOR) + return 0; + /* Create COLOR_PIPELINE property and attach */ drm_plane_create_color_pipeline_property(plane, pipelines, len); OpenPGP_0xE3520CC91929C8E7.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: [PATCH V8 03/43] drm/doc/rfc: Describe why prescriptive color pipeline is needed
You are right. Sorry for the noise. 在 2025/4/1 0:41, Alex Hung 写道: On 3/31/25 10:24, Shengyu Qu wrote: 在 2025/3/27 7:46, Alex Hung 写道: From: Harry Wentland Add documentation for color pipeline API. Signed-off-by: Alex Hung Signed-off-by: Harry Wentland --- v8: - Fix typo "definint" -> "defining" v7: - Add a commit messages v5: - Don't require BYPASS to succeed (Sebastian) - use DATA for 1D and 3D LUT types (Sebastian) - update 3DLUT ops to use 3DLUT_MODES and 3DLUT_MODE_INDEX - Add section on drm_colorop extensibility - Add color_pipeline.rst to RFC toc tree v4: - Drop IOCTL docs since we dropped the IOCTLs (Pekka) - Clarify reading and setting of COLOR_PIPELINE prop (Pekka) - Add blurb about not requiring to reject a pipeline due to incompatible ops, as long as op can be bypassed (Pekka) - Dropped informational strings (such as input CSC) as they're not actually intended to be advertised (Pekka) v3: - Describe DRM_CLIENT_CAP_PLANE_COLOR_PIPELINE (Sebastian) - Ask for clear documentation of colorop behavior (Sebastian) v2: - Update colorop visualizations to match reality (Sebastian, Alex Hung) - Updated wording (Pekka) - Change BYPASS wording to make it non-mandatory (Sebastian) - Drop cover-letter-like paragraph from COLOR_PIPELINE Plane Property section (Pekka) - Use PQ EOTF instead of its inverse in Pipeline Programming example (Melissa) - Add "Driver Implementer's Guide" section (Pekka) - Add "Driver Forward/Backward Compatibility" section (Sebastian, Pekka) Documentation/gpu/rfc/color_pipeline.rst | 378 +++ Documentation/gpu/rfc/index.rst | 3 + 2 files changed, 381 insertions(+) create mode 100644 Documentation/gpu/rfc/color_pipeline.rst diff --git a/Documentation/gpu/rfc/color_pipeline.rst b/ Documentation/ gpu/rfc/color_pipeline.rst new file mode 100644 index ..58bcc2a5ffd8 --- /dev/null +++ b/Documentation/gpu/rfc/color_pipeline.rst @@ -0,0 +1,378 @@ +.. SPDX-License-Identifier: GPL-2.0 + + +Linux Color Pipeline API + + +What problem are we solving? + + +We would like to support pre-, and post-blending complex color +transformations in display controller hardware in order to allow for +HW-supported HDR use-cases, as well as to provide support to +color-managed applications, such as video or image editors. + +It is possible to support an HDR output on HW supporting the Colorspace +and HDR Metadata drm_connector properties, but that requires the +compositor or application to render and compose the content into one +final buffer intended for display. Doing so is costly. + +Most modern display HW offers various 1D LUTs, 3D LUTs, matrices, and other +operations to support color transformations. These operations are often +implemented in fixed-function HW and therefore much more power efficient than +performing similar operations via shaders or CPU. + +We would like to make use of this HW functionality to support complex color +transformations with no, or minimal CPU or shader load. + + +How are other OSes solving this problem? + + +The most widely supported use-cases regard HDR content, whether video or +gaming. + +Most OSes will specify the source content format (color gamut, encoding transfer +function, and other metadata, such as max and average light levels) to a driver. +Drivers will then program their fixed-function HW accordingly to map from a +source content buffer's space to a display's space. + +When fixed-function HW is not available the compositor will assemble a shader to +ask the GPU to perform the transformation from the source content format to the +display's format. + +A compositor's mapping function and a driver's mapping function are usually +entirely separate concepts. On OSes where a HW vendor has no insight into +closed-source compositor code such a vendor will tune their color management +code to visually match the compositor's. On other OSes, where both mapping +functions are open to an implementer they will ensure both mappings match. + +This results in mapping algorithm lock-in, meaning that no-one alone can +experiment with or introduce new mapping algorithms and achieve +consistent results regardless of which implementation path is taken. + +Why is Linux different? +=== + +Unlike other OSes, where there is one compositor for one or more drivers, on +Linux we have a many-to-many relationship. Many compositors; many drivers. +In addition each compositor vendor or community has their own view of how +color management should be done. This is what makes Linux so beautiful. + +This means that a HW vendor can now no longer tune their driver to one +compositor, as tuning it to one could make it look f
Re: [PATCH V8 03/43] drm/doc/rfc: Describe why prescriptive color pipeline is needed
在 2025/3/27 7:46, Alex Hung 写道: From: Harry Wentland Add documentation for color pipeline API. Signed-off-by: Alex Hung Signed-off-by: Harry Wentland --- v8: - Fix typo "definint" -> "defining" v7: - Add a commit messages v5: - Don't require BYPASS to succeed (Sebastian) - use DATA for 1D and 3D LUT types (Sebastian) - update 3DLUT ops to use 3DLUT_MODES and 3DLUT_MODE_INDEX - Add section on drm_colorop extensibility - Add color_pipeline.rst to RFC toc tree v4: - Drop IOCTL docs since we dropped the IOCTLs (Pekka) - Clarify reading and setting of COLOR_PIPELINE prop (Pekka) - Add blurb about not requiring to reject a pipeline due to incompatible ops, as long as op can be bypassed (Pekka) - Dropped informational strings (such as input CSC) as they're not actually intended to be advertised (Pekka) v3: - Describe DRM_CLIENT_CAP_PLANE_COLOR_PIPELINE (Sebastian) - Ask for clear documentation of colorop behavior (Sebastian) v2: - Update colorop visualizations to match reality (Sebastian, Alex Hung) - Updated wording (Pekka) - Change BYPASS wording to make it non-mandatory (Sebastian) - Drop cover-letter-like paragraph from COLOR_PIPELINE Plane Property section (Pekka) - Use PQ EOTF instead of its inverse in Pipeline Programming example (Melissa) - Add "Driver Implementer's Guide" section (Pekka) - Add "Driver Forward/Backward Compatibility" section (Sebastian, Pekka) Documentation/gpu/rfc/color_pipeline.rst | 378 +++ Documentation/gpu/rfc/index.rst | 3 + 2 files changed, 381 insertions(+) create mode 100644 Documentation/gpu/rfc/color_pipeline.rst diff --git a/Documentation/gpu/rfc/color_pipeline.rst b/Documentation/gpu/rfc/color_pipeline.rst new file mode 100644 index ..58bcc2a5ffd8 --- /dev/null +++ b/Documentation/gpu/rfc/color_pipeline.rst @@ -0,0 +1,378 @@ +.. SPDX-License-Identifier: GPL-2.0 + + +Linux Color Pipeline API + + +What problem are we solving? + + +We would like to support pre-, and post-blending complex color +transformations in display controller hardware in order to allow for +HW-supported HDR use-cases, as well as to provide support to +color-managed applications, such as video or image editors. + +It is possible to support an HDR output on HW supporting the Colorspace +and HDR Metadata drm_connector properties, but that requires the +compositor or application to render and compose the content into one +final buffer intended for display. Doing so is costly. + +Most modern display HW offers various 1D LUTs, 3D LUTs, matrices, and other +operations to support color transformations. These operations are often +implemented in fixed-function HW and therefore much more power efficient than +performing similar operations via shaders or CPU. + +We would like to make use of this HW functionality to support complex color +transformations with no, or minimal CPU or shader load. + + +How are other OSes solving this problem? + + +The most widely supported use-cases regard HDR content, whether video or +gaming. + +Most OSes will specify the source content format (color gamut, encoding transfer +function, and other metadata, such as max and average light levels) to a driver. +Drivers will then program their fixed-function HW accordingly to map from a +source content buffer's space to a display's space. + +When fixed-function HW is not available the compositor will assemble a shader to +ask the GPU to perform the transformation from the source content format to the +display's format. + +A compositor's mapping function and a driver's mapping function are usually +entirely separate concepts. On OSes where a HW vendor has no insight into +closed-source compositor code such a vendor will tune their color management +code to visually match the compositor's. On other OSes, where both mapping +functions are open to an implementer they will ensure both mappings match. + +This results in mapping algorithm lock-in, meaning that no-one alone can +experiment with or introduce new mapping algorithms and achieve +consistent results regardless of which implementation path is taken. + +Why is Linux different? +=== + +Unlike other OSes, where there is one compositor for one or more drivers, on +Linux we have a many-to-many relationship. Many compositors; many drivers. +In addition each compositor vendor or community has their own view of how +color management should be done. This is what makes Linux so beautiful. + +This means that a HW vendor can now no longer tune their driver to one +compositor, as tuning it to one could make it look fairly different from +another compositor's color mapping. + +We need a better solution. + + +Descriptive API +=== + +An API that describes the source and destination colorspaces is a descriptive +API. It describes the input
Re: [PATCH V8 24/43] drm/amd/display: Skip color pipeline initialization for cursor plane
So currently we have to hope the compositor won't use DRM_PLANE_TYPE_CURSOR planes at all Why do we still register DRM_PLANE_TYPE_CURSOR in the driver? 在 2025/4/1 0:06, Alex Hung 写道: On 3/31/25 09:43, Shengyu Qu wrote: Hi, Thanks for reply. So currently we have to apply color conversion on the background plane of the cursor to do some color space conversion. What would happen if cursor and background plane needs different conversion config? Or we just give the cursor a dedicated plane? This scenario is not supported on AMD hardware, but software cursors on other plane types won't be affected. Best regards, Shengyu 在 2025/3/31 22:28, Alex Hung 写道: On 3/30/25 06:59, Shengyu Qu wrote: Hi, Do we really need to disable cursor plane color pipeline support? I don't think we need to disable that if it is supported, since there might be some user-defined colored cursor icon. This patch applies to AMD hardware only: https://elixir.bootlin.com/ linux/v6.13/source/Documentation/gpu/amdgpu/display/mpo- overview.rst#L101 Best regards, Shengyu For some unknown reason, seems my mail is not shown in the mail list archive, so I resent it. 在 2025/3/27 7:47, Alex Hung 写道: cursor plane does not need to have color pipeline. Signed-off-by: Alex Hung --- v7: - Add a commit messages drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/ amdgpu_dm_plane.c b/drivers/gpu/drm/amd/display/amdgpu_dm/ amdgpu_dm_plane.c index 9632b8b73e7e..b5b9b0b5da63 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c @@ -1792,6 +1792,9 @@ dm_plane_init_colorops(struct drm_plane *plane) struct drm_prop_enum_list pipelines[MAX_COLOR_PIPELINES]; int len = 0; + if (plane->type == DRM_PLANE_TYPE_CURSOR) + return 0; + /* Create COLOR_PIPELINE property and attach */ drm_plane_create_color_pipeline_property(plane, pipelines, len); OpenPGP_0xE3520CC91929C8E7.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: [PATCH V8 24/43] drm/amd/display: Skip color pipeline initialization for cursor plane
Hi, Thanks for reply. So currently we have to apply color conversion on the background plane of the cursor to do some color space conversion. What would happen if cursor and background plane needs different conversion config? Or we just give the cursor a dedicated plane? Best regards, Shengyu 在 2025/3/31 22:28, Alex Hung 写道: On 3/30/25 06:59, Shengyu Qu wrote: Hi, Do we really need to disable cursor plane color pipeline support? I don't think we need to disable that if it is supported, since there might be some user-defined colored cursor icon. This patch applies to AMD hardware only: https://elixir.bootlin.com/ linux/v6.13/source/Documentation/gpu/amdgpu/display/mpo-overview.rst#L101 Best regards, Shengyu For some unknown reason, seems my mail is not shown in the mail list archive, so I resent it. 在 2025/3/27 7:47, Alex Hung 写道: cursor plane does not need to have color pipeline. Signed-off-by: Alex Hung --- v7: - Add a commit messages drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c index 9632b8b73e7e..b5b9b0b5da63 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c @@ -1792,6 +1792,9 @@ dm_plane_init_colorops(struct drm_plane *plane) struct drm_prop_enum_list pipelines[MAX_COLOR_PIPELINES]; int len = 0; + if (plane->type == DRM_PLANE_TYPE_CURSOR) + return 0; + /* Create COLOR_PIPELINE property and attach */ drm_plane_create_color_pipeline_property(plane, pipelines, len); OpenPGP_0xE3520CC91929C8E7.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: [PATCH V8 24/43] drm/amd/display: Skip color pipeline initialization for cursor plane
在 2025/4/1 1:42, Alex Hung 写道: On 3/31/25 11:04, Shengyu Qu wrote: Or we can add some kind of "linked with" info to plane's COLOR_PIPELINE property, to let userspace know that cursor plane and background plane share the same colorop config. So that userspace could do extra conversion on cursor image data to avoid display wrong cursor color. That's over-complicate and makes little sense for both device drivers and userspace applications. If any planes share same colorop config, a device driver exposes the same color pipeline with the same colorops. If a plane does not support color pipeline or a driver doesn't want to support it, there is no color pipeline and no color objects. My understanding is that currently the driver would just report no colorop support on cursor plane and actually implement the background plane's colorop on cursor? Again it is up to compositors or apps to determine how color pipeline and colorops are used (or not). For example, both primary plane and overlay plane have the same color pipeline, HDR can be enabled only on overlay but not on primary. Still this is the cleanest way to let compositors know and deal with this special cursor plane behavior. Or if compositors want to use all planes with hw colorop + MPO(for power saving or sth.), they have to detect the gpu they are running on and apply a quirk for this. That's a "dirty" implementation. 在 2025/4/1 0:50, Shengyu Qu 写道: Thanks, I mistook about the MPO document. Maybe we should also disable colorop on the background plane of the cursor plane? So that compositors would do sw color convertion on both cursor plane and background plane, which should keep cursor display correctly. Cursor plane has no color pipeline and thus it has no colorop either. It inherits color processing from its parent plane. You can create a color pipeline for cursor plane for your hardware. If none of existing colorop matches your need, new colorop can be defined. 在 2025/4/1 0:34, Alex Hung 写道: On 3/31/25 10:31, Shengyu Qu wrote: Sorry for vague expression. I mean that I think we shouldn't register DRM_PLANE_TYPE_CURSOR in the driver, as we don't have actual hardware support. This is not true. AMD has hardware cursor support. 在 2025/4/1 0:26, Alex Hung 写道: On 3/31/25 10:12, Shengyu Qu wrote: So currently we have to hope the compositor won't use DRM_PLANE_TYPE_CURSOR planes at all Why do we still register DRM_PLANE_TYPE_CURSOR in the driver? I am not sure what your question is. A compositor can choose or skip any hardware features, but this discussion is out of the scope. 在 2025/4/1 0:06, Alex Hung 写道: On 3/31/25 09:43, Shengyu Qu wrote: Hi, Thanks for reply. So currently we have to apply color conversion on the background plane of the cursor to do some color space conversion. What would happen if cursor and background plane needs different conversion config? Or we just give the cursor a dedicated plane? This scenario is not supported on AMD hardware, but software cursors on other plane types won't be affected. Best regards, Shengyu 在 2025/3/31 22:28, Alex Hung 写道: On 3/30/25 06:59, Shengyu Qu wrote: Hi, Do we really need to disable cursor plane color pipeline support? I don't think we need to disable that if it is supported, since there might be some user-defined colored cursor icon. This patch applies to AMD hardware only: https:// elixir.bootlin.com/ linux/v6.13/source/Documentation/gpu/ amdgpu/ display/mpo- overview.rst#L101 Best regards, Shengyu For some unknown reason, seems my mail is not shown in the mail list archive, so I resent it. 在 2025/3/27 7:47, Alex Hung 写道: cursor plane does not need to have color pipeline. Signed-off-by: Alex Hung --- v7: - Add a commit messages drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/ amdgpu_dm_plane.c b/drivers/gpu/drm/amd/display/amdgpu_dm/ amdgpu_dm_plane.c index 9632b8b73e7e..b5b9b0b5da63 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c @@ -1792,6 +1792,9 @@ dm_plane_init_colorops(struct drm_plane *plane) struct drm_prop_enum_list pipelines[MAX_COLOR_PIPELINES]; int len = 0; + if (plane->type == DRM_PLANE_TYPE_CURSOR) + return 0; + /* Create COLOR_PIPELINE property and attach */ drm_plane_create_color_pipeline_property(plane, pipelines, len); OpenPGP_0xE3520CC91929C8E7.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: [PATCH V8 24/43] drm/amd/display: Skip color pipeline initialization for cursor plane
Or we can add some kind of "linked with" info to plane's COLOR_PIPELINE property, to let userspace know that cursor plane and background plane share the same colorop config. So that userspace could do extra conversion on cursor image data to avoid display wrong cursor color. 在 2025/4/1 0:50, Shengyu Qu 写道: Thanks, I mistook about the MPO document. Maybe we should also disable colorop on the background plane of the cursor plane? So that compositors would do sw color convertion on both cursor plane and background plane, which should keep cursor display correctly. 在 2025/4/1 0:34, Alex Hung 写道: On 3/31/25 10:31, Shengyu Qu wrote: Sorry for vague expression. I mean that I think we shouldn't register DRM_PLANE_TYPE_CURSOR in the driver, as we don't have actual hardware support. This is not true. AMD has hardware cursor support. 在 2025/4/1 0:26, Alex Hung 写道: On 3/31/25 10:12, Shengyu Qu wrote: So currently we have to hope the compositor won't use DRM_PLANE_TYPE_CURSOR planes at all Why do we still register DRM_PLANE_TYPE_CURSOR in the driver? I am not sure what your question is. A compositor can choose or skip any hardware features, but this discussion is out of the scope. 在 2025/4/1 0:06, Alex Hung 写道: On 3/31/25 09:43, Shengyu Qu wrote: Hi, Thanks for reply. So currently we have to apply color conversion on the background plane of the cursor to do some color space conversion. What would happen if cursor and background plane needs different conversion config? Or we just give the cursor a dedicated plane? This scenario is not supported on AMD hardware, but software cursors on other plane types won't be affected. Best regards, Shengyu 在 2025/3/31 22:28, Alex Hung 写道: On 3/30/25 06:59, Shengyu Qu wrote: Hi, Do we really need to disable cursor plane color pipeline support? I don't think we need to disable that if it is supported, since there might be some user-defined colored cursor icon. This patch applies to AMD hardware only: https:// elixir.bootlin.com/ linux/v6.13/source/Documentation/gpu/amdgpu/ display/mpo- overview.rst#L101 Best regards, Shengyu For some unknown reason, seems my mail is not shown in the mail list archive, so I resent it. 在 2025/3/27 7:47, Alex Hung 写道: cursor plane does not need to have color pipeline. Signed-off-by: Alex Hung --- v7: - Add a commit messages drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/ amdgpu_dm_plane.c b/drivers/gpu/drm/amd/display/amdgpu_dm/ amdgpu_dm_plane.c index 9632b8b73e7e..b5b9b0b5da63 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c @@ -1792,6 +1792,9 @@ dm_plane_init_colorops(struct drm_plane *plane) struct drm_prop_enum_list pipelines[MAX_COLOR_PIPELINES]; int len = 0; + if (plane->type == DRM_PLANE_TYPE_CURSOR) + return 0; + /* Create COLOR_PIPELINE property and attach */ drm_plane_create_color_pipeline_property(plane, pipelines, len); OpenPGP_0xE3520CC91929C8E7.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: [PATCH] Revert "drm/amd/display: Hardware cursor changes color when switched to software cursor"
Hi, Personally I prefer we always disable cursor HW degamma, and this is what color pipeline patch series have done for cursor plane(actually all colorops are disabled on cursor plane and its background plane). Since cursor plane shares colorops with cursor's background plane. Also, degamma a sRGB image with sRGB curve is not a good choice, or we have to fake screen TRC to sRGB curve while the screen actually is calibrated to power 2.2 curve. Best regards, Shengyu 在 2025/4/22 22:58, Melissa Wen 写道: This reverts commit 272e6aab14bbf98d7a06b2b1cd6308a02d4a10a1. Applying degamma curve to the cursor by default breaks Linux userspace expectation. On Linux, AMD display manager enables cursor degamma ROM just for implict sRGB on HW versions where degamma is split into two blocks: degamma ROM for pre-defined TFs and `gamma correction` for user/custom curves, and degamma ROM settings doesn't apply to cursor plane. Link: https://gitlab.freedesktop.org/drm/amd/-/issues/1513 Link: https://gitlab.freedesktop.org/drm/amd/-/issues/2803 Closes: https://gitlab.freedesktop.org/drm/amd/-/issues/4144 Reported-by: Michel Dänzer Signed-off-by: Melissa Wen --- Hi, I suspect there is a conflict of interest between OSes here, because this is not the first time this mechanism has been removed from the DC shared-code and after reintroduced [1]. I'd suggest that other OSes set the `dc_cursor_attributes attribute_flags.bits.ENABLE_CURSOR_DEGAMMA` to true by default, rather than removing the mechanism that is valid for the Linux driver. Similar to what the Linux AMD DM does for the implicit sRGB [2][3], but in their case, they just need to initialize with 1. Finally, thanks Michel for pointing this issue out to me and noticing the similarity to previous solution. [1] https://gitlab.freedesktop.org/agd5f/linux/-/commit/d9fbd64e8e317 [2] https://gitlab.freedesktop.org/agd5f/linux/-/commit/857b835f [3] https://gitlab.freedesktop.org/agd5f/linux/-/commit/66eba12a Best Regards, Melissa drivers/gpu/drm/amd/display/dc/dpp/dcn401/dcn401_dpp_cm.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/amd/display/dc/dpp/dcn401/dcn401_dpp_cm.c b/drivers/gpu/drm/amd/display/dc/dpp/dcn401/dcn401_dpp_cm.c index 1236e0f9a256..712aff7e17f7 100644 --- a/drivers/gpu/drm/amd/display/dc/dpp/dcn401/dcn401_dpp_cm.c +++ b/drivers/gpu/drm/amd/display/dc/dpp/dcn401/dcn401_dpp_cm.c @@ -120,10 +120,11 @@ void dpp401_set_cursor_attributes( enum dc_cursor_color_format color_format = cursor_attributes->color_format; int cur_rom_en = 0; - // DCN4 should always do Cursor degamma for Cursor Color modes if (color_format == CURSOR_MODE_COLOR_PRE_MULTIPLIED_ALPHA || color_format == CURSOR_MODE_COLOR_UN_PRE_MULTIPLIED_ALPHA) { - cur_rom_en = 1; + if (cursor_attributes->attribute_flags.bits.ENABLE_CURSOR_DEGAMMA) { + cur_rom_en = 1; + } } REG_UPDATE_3(CURSOR0_CONTROL, OpenPGP_0xE3520CC91929C8E7.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature