[darktable-dev] Local Contrast Module broken using OpenCL
I recently migrated to a new Linux workstation with OpenCL capability, and Darktable 3.0.2 from the OpenSUSE 15.1 distro repository. The Local Contrast Module (LC) is broken and produces ugly, extreme 'shading' and artifacts in local laplacian mode that makes it unusable. The problem manifests on viewing images previously developed on my old laptop with LC enabled (same DT version but no OpenCL) as well as developing on the workstation. Everything else in DT (and other apps) that I use in my workflow is working as normal on the workstation. OpenCL compiled and loaded with no errors. There are no runtime error messages, normal or with debug. Disabling OpenCL fixes the Local Contrast module and it works as expected. I've tried rebuilding locallaplacian.cl.bin, which again compiled and loaded without error, but the problem remains. Appears to a be bug in OpenCL code. Any ideas/help? Thanks Workstation info: System: Host: photoworkstation Kernel: 5.7.6-2.g1549350-default x86_64 bits: 64 compiler: gcc v: 10.1.1 Desktop: Gnome 3.26.2 wm: gnome-shell dm: GDM 3.26.2.1 Distro: openSUSE Leap 15.1 Machine: Type: Desktop Mobo: ASRock model: X570 Phantom Gaming 4, PCIe 4.0 UEFI: American Megatrends v: P2.30 date: 01/13/2020 CPU: Topology: 12-Core model: AMD Ryzen 9 3900X bits: 64 type: MT MCP arch: Zen 2 L2 cache: 6144 KiB flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm bogomips: 182399 Speed: 2206 MHz min/max: 2200/3800 MHz boost: enabled Core speeds (MHz): 1: 2200 2: 2199 3: 3599 4: 2058 5: 2200 6: 2199 7: 2193 8: 3593 9: 3595 10: 3589 11: 2794 12: 1866 13: 2199 14: 1860 15: 3600 16: 3595 17: 2059 18: 3595 19: 3604 20: 3600 21: 2795 22: 1866 23: 1866 24: 3597 Memory: Dual-channel 32Gb, 2 x 16Gb DDR4 3200MHz, CAS=14, TEAMGROUP 8 Kit, Samsung B-Die Graphics: Device-1: Advanced Micro Devices [AMD/ATI] Navi 14 [Radeon RX 5500/5500M / Pro 5500M] vendor: Micro-Star MSI driver: amdgpu v: kernel bus ID: 0b:00.0 chip ID: 1002:7340 Display: x11 server: X.Org 1.20.3 compositor: gnome-shell driver: amdgpu,ati unloaded: fbdev,modesetting,radeon,vesa resolution: 1: 2560x1440~60Hz 2: 3840x2160~60Hz s-dpi: 96 OpenGL: renderer: llvmpipe (LLVM 7.0 128 bits) v: 3.3 Mesa 18.3.2 compat-v: 3.1 direct render: Yes darktable --version this is darktable 3.0.2 compile options: bit depth is 64 bit normal build SSE2 optimized codepath enabled OpenMP support enabled OpenCL support enabled Lua support enabled, API version 5.0.2 Colord support enabled gPhoto2 support enabled GraphicsMagick support enabled OpenEXR support enabled OpenCL: installed from AMD AMDGPU-PRO package [opencl_init] device 0: gfx1012 GLOBAL_MEM_SIZE: 8176MB MAX_WORK_GROUP_SIZE: 256 MAX_WORK_ITEM_DIMENSIONS: 3 MAX_WORK_ITEM_SIZES: [ 1024 1024 1024 ] DRIVER_VERSION: 3075.10 (PAL,LC) DEVICE_VERSION: OpenCL 2.0 AMD-APP (3075.10) The OpenCL build log shows no errors. ___ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
Re: [darktable-dev] Local Contrast Module broken using OpenCL
Thanks for the prompt reply Andreas. What a pain, I use local laplacian a lot. The github threads say the AMDGPU-PRO drivers don't have this problem, so I'm wondering if I should replace the open source amdgpu driver with the PRO version as a workaround for the time being. Do you know if this is a realistic option for a non-dev user? On 13/08/2020 13:53, Andreas Schneider wrote: On Thursday, 13 August 2020 14:50:02 CEST Dusenberg wrote: I recently migrated to a new Linux workstation with OpenCL capability, and Darktable 3.0.2 from the OpenSUSE 15.1 distro repository. The Local Contrast Module (LC) is broken and produces ugly, extreme 'shading' and artifacts in local laplacian mode that makes it unusable. The problem manifests on viewing images previously developed on my old laptop with LC enabled (same DT version but no OpenCL) as well as developing on the workstation. Everything else in DT (and other apps) that I use in my workflow is working as normal on the workstation. OpenCL compiled and loaded with no errors. There are no runtime error messages, normal or with debug. Disabling OpenCL fixes the Local Contrast module and it works as expected. I've tried rebuilding locallaplacian.cl.bin, which again compiled and loaded without error, but the problem remains. Appears to a be bug in OpenCL code. Any ideas/help? ___ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
Re: [darktable-dev] Local Contrast Module broken using OpenCL
Thanks for your time Šarūnas and Germano. This has got me worried because my workstation has the OpenCL from the AMDGPU-PRO version installed (but nothing else from it) and I'm using the open source amdgpu driver package. My reading of the bug analyses done by the devs (https://github.com/RadeonOpenCompute/ROCm-OpenCL-Runtime/issues/103), is that I shouldn't be seeing the problem as it only affects the open source OpenCL package running with the open source amdgpu driver. Have I got this right? Is it possible I'm somehow using the open source OpenCL instead of the PRO? The output from darktable and inxi are below - do the experts know if this proves what I'm running? darktable -d opencl: [opencl_init] device 0: gfx1012 GLOBAL_MEM_SIZE: 8176MB MAX_WORK_GROUP_SIZE: 256 MAX_WORK_ITEM_DIMENSIONS: 3 MAX_WORK_ITEM_SIZES: [ 1024 1024 1024 ] DRIVER_VERSION: 3075.10 (PAL,LC) DEVICE_VERSION: OpenCL 2.0 AMD-APP (3075.10) inxi -Gxx: Graphics: Device-1: Advanced Micro Devices [AMD/ATI] Navi 14 [Radeon RX 5500/5500M / Pro 5500M] vendor: Micro-Star MSI driver: amdgpu v: kernel bus ID: 0b:00.0 chip ID: 1002:7340 Thanks On 14/08/2020 21:47, Šarūnas wrote: On 8/14/20 2:37 PM, Dusenberg wrote: Thanks for the prompt reply Andreas. What a pain, I use local laplacian a lot. The github threads say the AMDGPU-PRO drivers don't have this problem, so I'm wondering if I should replace the open source amdgpu driver with the PRO version as a workaround for the time being. Do you know if this is a realistic option for a non-dev user? If this is a problem with the open source ROCm OpenCL library, you can run open source amdgpu (present in Linux kernel) with OpenCL from AMDGPU-PRO. Here is for Ubuntu, but might be adaptable to others: https://math.dartmouth.edu/~sarunas/amdgpu.html On 14/08/2020 19:46, Germano Massullo wrote: Il 14/08/20 20:37, Dusenberg ha scritto: Thanks for the prompt reply Andreas. What a pain, I use local laplacian a lot. The github threads say the AMDGPU-PRO drivers don't have this problem, so I'm wondering if I should replace the open source amdgpu driver with the PRO version as a workaround for the time being. Do you know if this is a realistic option for a non-dev user? I everyday use amdgpu drivers, but when I use darktable, I use the proprietary amdgpu-pro OpenCL drivers. Here is my procedure: Just download the drivers for the distro that is more similar to yours (example, RPM or Deb packages). The example will follow RPM architecture, but for Debs the procedure is almost the same. Unpack the tar.xz file in a folder let's call it /home/user/unpacked After that, you will have folder /home/user/unpacked/amdgpu-pro that contains subfolders amdgpu-pro-install repodata RPMS SRPMS go in RPMS and take all RPM files that are in subfolder /home/user/unpacked/amdgpu-pro/x86_64 and put them together someelsewhere, for example /home/user/amd_opencl then unpack them all. You will get /home/user/amd_opencl/etc/ /home/user/amd_opencl/lib/ /home/user/amd_opencl/opt/ /home/user/amd_opencl/usr/ Then to run darktable OPENCL_VENDOR_PATH=/home/user/amd_opencl/etc/OpenCL/vendors/ LD_LIBRARY_PATH=/home/user/amd_opencl/opt/amdgpu-pro/lib64/ darktable ___ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org ___ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
Re: [darktable-dev] Local Contrast Module broken using OpenCL
@Sturm Flut: Thanks for clarifying. Now I know there's a solution, I can live with the slower performance of running OpenCL disabled on a new workstation while I work out how to make the switch from using the open source amdgpu driver to the AMDGPU-PRO driver. Regards On 16/08/2020 16:05, Sturm Flut wrote: Hi, Am 15.08.20 um 13:21 schrieb Dusenberg: [..] I shouldn't be seeing the problem as it only affects the/open source/ OpenCL package running with the/open source/ amdgpu driver. Have I got this right? Yes. There have been problems with ROCm, darktable and visual artifacts for years. AMDGPU-PRO works fine. Is it possible I'm somehow using the open source OpenCL instead of the PRO? No Linux distribution I know of uses ROCm by default, many (e.g. Ubuntu) don't even include it in the repositories. Getting ROCm-OpenCL running on your RX 5500 (Navi 14) is also not trivial. So ROCm is not something you would be running without knowing it. kind regards, Simon ___ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org ___ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
Re: [darktable-dev] Local Contrast Module broken using OpenCL
Ah, I didn't know that, thanks. I'll have a look at 3.2.1 and read the posts about it and then decide what to do. On 16/08/2020 17:27, Andreas Schneider wrote: On Friday, 14 August 2020 20:37:29 CEST Dusenberg wrote: Thanks for the prompt reply Andreas. There is a workaround for the issue in 3.2.1. So you can just use ROCm without any problems with darktable 3.2.1. ___ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
Re: [darktable-dev] Local Contrast Module broken using OpenCL
...and finally.
I tried installing the full AMDGPU-PRO suite but couldn't get the
desktop to start, and eventually I ran out of time so restored the
system back to where I'd started.
Later I installed darktable 3.2.1 and Local Contrast works
correctly with OpenCL from AMDGPU-PRO and open source amdgpu
drivers! Much simpler. Job done.
Thanks everyone.
Forwarded Message
Subject:
Re: [darktable-dev] Local Contrast Module broken using
OpenCL
Date:
Mon, 17 Aug 2020 13:15:02 +0100
From:
Dusenberg
To:
darktable-dev@lists.darktable.org
Ah, I didn't know that, thanks. I'll have a look
at 3.2.1 and read the posts about it and then decide what to do.
On 16/08/2020 17:27, Andreas
Schneider wrote:
On Friday, 14 August 2020 20:37:29 CEST Dusenberg wrote:
Thanks for the prompt reply Andreas.
There is a workaround for the issue in 3.2.1. So you can just use ROCm without
any problems with darktable 3.2.1.
___
darktable developer mailing list
to unsubscribe send a mail to
darktable-dev+unsubscr...@lists.darktable.org
___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
[darktable-dev] Help with lua export script please
Hi, I'm needing some expert help with my first attempt at a dt lua script. In the 'final output' stage of my workflow I create images in various reduced sizes and formats from developed raws mainly for placing into non-darktable managed libraries or to publish. I use imagemagik 'convert' to resize at the end of the shrink and reformat process, and exiftool to copy metadata from the tifs if required. At present this in a bash script outside of dt and is a fairly cumbersome procedure which I really want to automate. My idea is to create a new storage target in dt to replace the bash script. It has taken a long time but I've got the basic lua script together with code snips taken from the gimp.lua script, image_stack.lua, and others and I've been referring to the darktable lua api documentation. I've got the new gui widgets to display and pre-populate with defaults (well most of them) but I have lost the standard export options for export destination and filename, bit depth, and file format. The gimp.lua, and other scripts, automatically create 16bit tifs in /tmp without the user having to specify this, but I haven't been able to find out how this is done. Also I can't find how to access the standard export data fields. I don't want to test what happens when I press 'export' until I've got these basics correct. My immediate questions I need guidance with are: 1 - how do I ensure the export is always to /tmp, is 16-bit, and is in tif format? 2 - how do I ensure the exported tif files use my naming convention? 3 - how do I check if metadata was selected for export so I can determine if I need to run a metadata copy to the output files? Thanks in advance. ___ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
Re: [darktable-dev] Help with lua export script please
Thanks for your prompt reply Bill. I didn't spot
your extra dt lua scripts, so I'll have a look at postsharpen.lua
and work out what I need to do.
On 15/11/2020 00:53, William Ferguson
wrote:
On Sat, Nov 14, 2020 at
10:36 AM Dusenberg <dusenb...@gmx.co.uk> wrote:
Hi, I'm needing some expert help with
my first attempt at a dt lua script.
In the 'final output' stage of my workflow I create
images in various reduced sizes and formats from
developed raws mainly for placing into non-darktable
managed libraries or to publish. I use imagemagik
'convert' to resize at the end of the shrink and
reformat process, and exiftool to copy metadata from the
tifs if required. At present this in a bash script
outside of dt and is a fairly cumbersome procedure which
I really want to automate. My idea is to create a new
storage target in dt to replace the bash script.
It has taken a long time but I've got the basic lua
script together with code snips taken from the gimp.lua
script, image_stack.lua, and others and I've been
referring to the darktable lua api documentation. I've
got the new gui widgets to display and pre-populate with
defaults (well most of them) but I have lost the
standard
export options for export destination and filename, bit
depth, and file format. The gimp.lua, and other scripts,
automatically create 16bit tifs in /tmp without the user
having to specify this, but I haven't been able to find
out how this is done. Also I can't find how to access
the standard export data fields. I don't want to test what
happens when I press 'export' until I've got these
basics correct.
My immediate questions I need guidance with are:
1 - how do I ensure the export is always to /tmp, is
16-bit, and is in tif format?
The default destination is always
dt.configuration.config_dir, which on Linux and MacOS is
/tmp. You can manipulate the final destination of the
export files, if you want to.
The exporter "remembers" the last format you exported in,
so if you export a 16 bit tif, then the next time you open
the exporter that's what it will be set at. You can
restrict the
choices of file types to tif only with the following code
local function support_format(storage, format) --tells dt
we only support TIFF export type
local ret = false
local temp = string.find(format.name, 'TIFF')
if temp ~= nil then ret = true end
return ret
end
Specify the function support_format when you declare the
register_storage to implement it.
2 - how do I ensure the exported tif files use my naming
convention?
The exported files are sent to tmp as $(FILENAME).tif or
whatever export type you select. You can retrieve them from
there and name them anyway you want.
3 - how do I check if metadata was selected for export
so I can determine if I need to run a metadata copy to
the output files?
I would leave this up to the settings. If you want
metadata exported, then pick what you want to export and let
the exporter handle it. If you share the script and the
user doesn't want
to export metadata, but the script checks the file and
does it anyway, it might be a nasty surprise.
Thanks in advance.
I wrote a script, postsharpen.lua, that shows how to do
most of what you asked about. It exports files to /tmp, the
runs convert over them to sharpen them, then saves them to a
different location, with possibly a different name
[darktable-dev] Duplicated version number created for existing image
dt 4.2.1 (OBS), Linux Mint 21,Ubuntu 22.04 jammy I have an image from March 2020 developed in darktable. I went back to it today to try another edit on it (its a monochrome rendition that I just can't get 'right'). However, today when I created a duplicate of this 2020 image in dt 4.2.1, it was given version number '3' - which already exists for that image (there are seven pre-existing duplicates). I see that dt has also given the new duplicate a different 'image id' to the original RAW image. I've never seen this before, although its not often I go back in time like this. My workflow is that I always create a new version (duplicate) of the base RAW for a different edit so I can trace back any final output that may result. My filenaming system is '___' where filename is composed of ''. Original camera images are renamed during download onto my workstation via a bespoke script (ie outside dt). I use variables in the dt export module to ensure any output follows this format. This provides unique identification of every image and its derivatives across my libraries, even when intermediate tiffs are involved in say, focus stacks. This is critical for me - I can't have two different edits of a RAW with the same filename! Why has it happened and what can I do about it? Thanks ___ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
