Thank you for this study Aurélien As far as I understand, TV and Laplacians are complementary as they detect noise in different regions of the image (noise in sharp edge for Laplacian, noise elsewhere for TV). Though, I do not understand why you multiply the TV and Laplacian results to get the mask. Multiplying them would result in a mask containing non-zero values only for pixels that are detected as noise both by TV and Laplacian. Is there a particular reason for multiplying (or did I misunderstood something?), or could we take the maximum value among TV and Laplacian for each pixel instead?
Thanks again Cheers, rawfiner 2018-07-01 3:45 GMT+02:00 Aurélien Pierre <rese...@aurelienpierre.com>: > Hi, > > I have done experiments on that matter and took the opportunity to > correct/test further my code. > > So here are my attempts to code a noise mask and a sharpness mask with > total variation and laplacian norms : https://github.com/ > aurelienpierre/Image-Cases-Studies/blob/master/notebooks/ > Total%20Variation%20masking.ipynb > > Performance benchmarks are at the end. > > Cheers, > > Aurélien. > > Le 17/06/2018 à 15:03, rawfiner a écrit : > > > > Le dimanche 17 juin 2018, Aurélien Pierre <rese...@aurelienpierre.com> a > écrit : > >> >> >> Le 13/06/2018 à 17:31, rawfiner a écrit : >> >> >> >> Le mercredi 13 juin 2018, Aurélien Pierre <rese...@aurelienpierre.com> a >> écrit : >> >>> >>> >>>> On Thu, Jun 14, 2018 at 12:23 AM, Aurélien Pierre >>>> <rese...@aurelienpierre.com> wrote: >>>> > Hi, >>>> > >>>> > The problem of a 2-passes denoising method involving 2 differents >>>> > algorithms, the later applied where the former failed, could be the >>>> grain >>>> > structure (the shape of the noise) would be different along the >>>> picture, >>>> > thus very unpleasing. >>> >>> >>> I agree that the grain structure could be different. Indeed, the grain >>> could be different, but my feeling (that may be wrong) is that it would be >>> still better than just no further processing, that leaves some pixels >>> unprocessed (they could form grain structures far from uniform if we are >>> not lucky). >>> If you think it is only due to a change of algorithm, I guess we could >>> apply non local means again on pixels where a first pass failed, but with >>> different parameters to be quite confident that the second pass will work. >>> >>> That sounds better to me… but practice will have the last word. >>> >> >> Ok :-) >> >>> >>> >>>> > >>>> > I thought maybe we could instead create some sort of total variation >>>> > threshold on other denoising modules : >>>> > >>>> > compute the total variation of each channel of each pixel as the >>>> divergence >>>> > divided by the L1 norm of the gradient - we then obtain a "heatmap" >>>> of the >>>> > gradients over the picture (contours and noise) >>>> > let the user define a total variation threshold and form a mask where >>>> the >>>> > weights above the threshold are the total variation and the weights >>>> below >>>> > the threshold are zeros (sort of a highpass filter actually) >>>> > apply the bilateral filter according to this mask. >>>> > >>>> > This way, if the user wants to stack several denoising modules, he >>>> could >>>> > protect the already-cleaned areas from further denoising. >>>> > >>>> > What do you think ? >>> >>> >>> That sounds interesting. >>> This would maybe allow to keep some small variations/details that are >>> not due to noise or not disturbing, while denoising the other parts. >>> Also, it may be computationally interesting (depends on the complexity >>> of the total variation computation, I don't know it), as it could reduce >>> the number of pixels to process. >>> I guess the user could use something like that also the other way?: to >>> protect high detailed zones and apply the denoising on quite smoothed zones >>> only, in order to be able to use stronger denoising on zones that are >>> supposed to be background blur. >>> >>> >>> The noise is high frequency, so the TV (total variation) threshold will >>> have to be high pass only. The hypothesis behind the TV thresholding is >>> noisy pixels should have abnormally higher gradients than true details, so >>> you isolate them this way. Selecting noise in low frequencies areas would >>> require in addition something like a guided filter, which I believe is what >>> is used in the dehaze module. The complexity of the TV computation depends >>> on the order of accuracy you expect. >>> >>> A classic approximation of the gradient is using a convolution product >>> with Sobel or Prewitt operators (3×3 arrays, very efficient, fairly >>> accurate for edges, probably less accurate for punctual noise). I have >>> developped myself optimized methods using 2, 4, and 8 neighbouring pixels >>> that give higher order accuracy, given the sparsity of the data, at the >>> expense of computing cost : https://github.com/aurelienpie >>> rre/Image-Cases-Studies/blob/947fd8d5c2e4c3384c80c1045d86f8c >>> f89ddcc7e/lib/deconvolution.pyx#L342 (ignore the variable ut in the >>> code, only u is relevant for us here). >>> >> Great, thanks for the explanations. >> Looking at the code of the 8 neighbouring pixels, I wonder if we would >> make sense to compute something like that on raw data considering only >> neighbouring pixels of the same color? >> >> >> the RAW data are even more sparse, so the gradient can't be computed this >> way. One would have to tweak the Taylor theorem to find an expression of >> gradient for sparse data. And that would be different for Bayer and X-Trans >> patterns. It's a bit of a conundrum. >> > > Ok, thank you for these explainations > > >> >> Also, when talking about the mask formed from the heat map, do you mean >> that the "heat" would give for each pixel a weight to use between input and >> output? (i.e. a mask that is not only ones and zeros, but that controls how >> much input and output are used for each pixel) >> If so, I think it is a good idea to explore! >> >> yes, exactly, think of it as an opacity mask where you remap the >> user-input TV threshold and the lower values to 0, the max magnitude of TV >> to 1, and all the values in between accordingly. >> > > Ok that is really cool! It seems a good idea to try to use that! > > rawfiner > > >> >> >> rawfiner >> >>> >>> >>> >>>> > >>>> > Aurélien. >>>> > >>>> > >>>> > Le 13/06/2018 à 03:16, rawfiner a écrit : >>>> > >>>> > Hi, >>>> > >>>> > I don't have the feeling that increasing K is the best way to improve >>>> noise >>>> > reduction anymore. >>>> > I will upload the raw next week (if I don't forget to), as I am not >>>> at home >>>> > this week. >>>> > My feeling is that doing non local means on raw data gives much bigger >>>> > improvement than that. >>>> > I still have to work on it yet. >>>> > I am currently testing some raw downsizing ideas to allow a fast >>>> execution >>>> > of the algorithm. >>>> > >>>> > Apart of that, I also think that to improve noise reduction such as >>>> the >>>> > denoise profile in nlm mode and the denoise non local means, we could >>>> do a 2 >>>> > passes algorithm, with non local means applied first, and then a >>>> bilateral >>>> > filter (or median filter or something else) applied only on pixels >>>> where non >>>> > local means failed to find suitable patches (i.e. pixels where the >>>> sum of >>>> > weights was close to 0). >>>> > The user would have a slider to adjust this setting. >>>> > I think that it would make easier to have a "uniform" output (i.e. an >>>> output >>>> > where noise has been reduced quite uniformly) >>>> > I have not tested this idea yet. >>>> > >>>> > Cheers, >>>> > rawfiner >>>> > >>>> > Le lundi 11 juin 2018, johannes hanika <hana...@gmail.com> a écrit : >>>> >> >>>> >> hi, >>>> >> >>>> >> i was playing with noise reduction presets again and tried the large >>>> >> neighbourhood search window. on my shots i could very rarely spot a >>>> >> difference at all increasing K above 7, and even less so going above >>>> >> 10. the image you posted earlier did show quite a substantial >>>> >> improvement however. i was wondering whether you'd be able to share >>>> >> the image so i can evaluate on it? maybe i just haven't found the >>>> >> right test image yet, or maybe it's camera dependent? >>>> >> >>>> >> (and yes, automatic and adaptive would be better but if we can ship a >>>> >> simple slider that can improve matters, maybe we should) >>>> >> >>>> >> cheers, >>>> >> jo >>>> >> >>>> >> >>>> >> >>>> >> On Mon, Jan 29, 2018 at 2:05 AM, rawfiner <rawfi...@gmail.com> >>>> wrote: >>>> >> > Hi >>>> >> > >>>> >> > Yes, the patch size is set to 1 from the GUI, so it is not a >>>> bilateral >>>> >> > filter, and I guess it corresponds to a patch window size of 3x3 >>>> in the >>>> >> > code. >>>> >> > The runtime difference is near the expected quadratic slowdown: >>>> >> > 1,460 secs (8,379 CPU) for 7 and 12,794 secs (85,972 CPU) for 25, >>>> which >>>> >> > means about 10.26x slowdown >>>> >> > >>>> >> > If you want to make your mind on it, I have pushed a branch here >>>> that >>>> >> > integrates the K parameter in the GUI: >>>> >> > https://github.com/rawfiner/darktable.git >>>> >> > The branch is denoise-profile-GUI-K >>>> >> > >>>> >> > I think that it may be worth to see if an automated approach for >>>> the >>>> >> > choice >>>> >> > of K may work, in order not to integrate the parameter in the GUI. >>>> >> > I may try to implement the approach of Kervann and Boulanger (the >>>> >> > reference >>>> >> > from the darktable blog post) to see how it performs. >>>> >> > >>>> >> > cheers, >>>> >> > rawfiner >>>> >> > >>>> >> > >>>> >> > 2018-01-27 13:50 GMT+01:00 johannes hanika <hana...@gmail.com>: >>>> >> >> >>>> >> >> heya, >>>> >> >> >>>> >> >> thanks for the reference! interesting interpretation how the >>>> blotches >>>> >> >> form. not sure i'm entirely convinced by that argument. >>>> >> >> your image does look convincing though. let me get this right.. >>>> you >>>> >> >> ran with radius 1 which means patch window size 3x3? not 1x1 which >>>> >> >> would be a bilateral filter effectively? >>>> >> >> >>>> >> >> also what was the run time difference? is it near the expected >>>> >> >> quadratic slowdown from 7 (i.e. 15x15) to 25 (51x51) so about >>>> 11.56x >>>> >> >> slower with the large window size? (test with darktable -d perf) >>>> >> >> >>>> >> >> since nlmeans isn't the fastest thing, even with this coalesced >>>> way of >>>> >> >> implementing it, we should certainly keep an eye on this. >>>> >> >> >>>> >> >> that being said if we can often times get much better results we >>>> >> >> should totally expose this in the gui, maybe with a big warning >>>> that >>>> >> >> it really severely impacts speed. >>>> >> >> >>>> >> >> cheers, >>>> >> >> jo >>>> >> >> >>>> >> >> On Sat, Jan 27, 2018 at 7:34 AM, rawfiner <rawfi...@gmail.com> >>>> wrote: >>>> >> >> > Thank you for your answer >>>> >> >> > I perfectly agree with the fact that the GUI should not become >>>> >> >> > overcomplicated. >>>> >> >> > >>>> >> >> > As far as I understand, the pixels within a small zone may >>>> suffer >>>> >> >> > from >>>> >> >> > correlated noise, and there is a risk of noise to noise >>>> matching. >>>> >> >> > That's why this paper suggest not to take pixels that are too >>>> close >>>> >> >> > to >>>> >> >> > the >>>> >> >> > zone we are correcting, but to take them a little farther (see >>>> the >>>> >> >> > caption >>>> >> >> > of Figure 2 for a quick explaination): >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> > https://pdfs.semanticscholar.org/c458/71830cf535ebe6c2b7656f >>>> 6a205033761fc0.pdf >>>> >> >> > (in case you ask, unfortunately there is a patent associated >>>> with >>>> >> >> > this >>>> >> >> > approach, so we cannot implement it) >>>> >> >> > >>>> >> >> > Increasing the neighborhood parameter results in having >>>> >> >> > proportionally >>>> >> >> > less >>>> >> >> > problem of correlation between surrounding pixels, and >>>> decreases the >>>> >> >> > size of >>>> >> >> > the visible spots. >>>> >> >> > See for example the two attached pictures: one with size 1, >>>> force 1, >>>> >> >> > and >>>> >> >> > K 7 >>>> >> >> > and the other with size 1, force 1, and K 25. >>>> >> >> > >>>> >> >> > I think that the best would probably be to adapt K >>>> automatically, in >>>> >> >> > order >>>> >> >> > not to affect the GUI, and as we may have different levels of >>>> noise >>>> >> >> > in >>>> >> >> > different parts of an image. >>>> >> >> > In this post >>>> >> >> > >>>> >> >> > (https://www.darktable.org/2012/12/profiling-sensor-and-phot >>>> on-noise/), >>>> >> >> > this >>>> >> >> > paper is cited: >>>> >> >> > >>>> >> >> > [4] charles kervrann and jerome boulanger: optimal spatial >>>> adaptation >>>> >> >> > for >>>> >> >> > patch-based image denoising. ieee trans. image process. vol. >>>> 15, no. >>>> >> >> > 10, >>>> >> >> > 2006 >>>> >> >> > >>>> >> >> > As far as I understand, it gives a way to choose an adaptated >>>> window >>>> >> >> > size >>>> >> >> > for each pixel, but I don't see in the code anything related to >>>> that >>>> >> >> > >>>> >> >> > Maybe is this paper related to the TODOs in the code ? >>>> >> >> > >>>> >> >> > Was it planned to implement such a variable window approach ? >>>> >> >> > >>>> >> >> > Or if it is already implemented, could you point me where ? >>>> >> >> > >>>> >> >> > Thank you >>>> >> >> > >>>> >> >> > rawfiner >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> > 2018-01-26 9:05 GMT+01:00 johannes hanika <hana...@gmail.com>: >>>> >> >> >> >>>> >> >> >> hi, >>>> >> >> >> >>>> >> >> >> if you want, absolutely do play around with K. in my tests it >>>> did >>>> >> >> >> not >>>> >> >> >> lead to any better denoising. to my surprise a larger K often >>>> led to >>>> >> >> >> worse results (for some reason often the relevance of >>>> discovered >>>> >> >> >> patches decreases with distance from the current point). >>>> that's why >>>> >> >> >> K >>>> >> >> >> is not exposed in the gui, no need for another irrelevant and >>>> >> >> >> cryptic >>>> >> >> >> parameter. if you find a compelling case where this indeed >>>> leads to >>>> >> >> >> better denoising we could rethink that. >>>> >> >> >> >>>> >> >> >> in general NLM is a 0-th order denoising scheme, meaning the >>>> prior >>>> >> >> >> is >>>> >> >> >> piecewise constant (you claim the pixels you find are trying to >>>> >> >> >> express /the same/ mean, so you average them). if you let that >>>> >> >> >> algorithm do what it would really like to, it'll create >>>> unpleasant >>>> >> >> >> blotches of constant areas. so for best results we need to >>>> tone it >>>> >> >> >> down one way or another. >>>> >> >> >> >>>> >> >> >> cheers, >>>> >> >> >> jo >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> On Fri, Jan 26, 2018 at 7:36 AM, rawfiner <rawfi...@gmail.com> >>>> >> >> >> wrote: >>>> >> >> >> > Hi >>>> >> >> >> > >>>> >> >> >> > I am surprised to see that we cannot control the neighborhood >>>> >> >> >> > parameter >>>> >> >> >> > for >>>> >> >> >> > the NLM algorithm (neither for the denoise non local mean, >>>> nor for >>>> >> >> >> > the >>>> >> >> >> > denoise profiled) from the GUI. >>>> >> >> >> > I see in the code (denoiseprofile.c) this TODO that I don't >>>> >> >> >> > understand: >>>> >> >> >> > "// >>>> >> >> >> > TODO: fixed K to use adaptive size trading variance and >>>> bias!" >>>> >> >> >> > And just some lines after that: "// TODO: adaptive K tests >>>> here!" >>>> >> >> >> > (K is the neighborhood parameter of the NLM algorithm). >>>> >> >> >> > >>>> >> >> >> > In practice, I think that being able to change the >>>> neighborhood >>>> >> >> >> > parameter >>>> >> >> >> > allows to have a better noise reduction for one image. >>>> >> >> >> > For example, choosing a bigger K allows to reduce the >>>> spotted >>>> >> >> >> > aspect >>>> >> >> >> > that >>>> >> >> >> > one can get on high ISO images. >>>> >> >> >> > >>>> >> >> >> > Of course, increasing K increase computational time, but I >>>> think >>>> >> >> >> > we >>>> >> >> >> > could >>>> >> >> >> > find an acceptable range that would still be useful. >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> > Is there any reason for not letting the user control the >>>> >> >> >> > neighborhood >>>> >> >> >> > parameter in the GUI ? >>>> >> >> >> > Also, do you understand the TODOs ? >>>> >> >> >> > I feel that we would probably get better denoising by fixing >>>> >> >> >> > these, >>>> >> >> >> > but >>>> >> >> >> > I >>>> >> >> >> > don't understand them. >>>> >> >> >> > >>>> >> >> >> > I can spend some time on these TODOs, or to add the K >>>> parameter to >>>> >> >> >> > the >>>> >> >> >> > interface if you think it is worth it (I think so but it is >>>> only >>>> >> >> >> > my >>>> >> >> >> > personal >>>> >> >> >> > opinion), but I have to understand what the TODOs mean before >>>> >> >> >> > >>>> >> >> >> > Thank you for your help >>>> >> >> >> > >>>> >> >> >> > rawfiner >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> > ____________________________________________________________ >>>> _______________ >>>> >> >> >> > darktable developer mailing list to unsubscribe send a mail >>>> to >>>> >> >> >> > darktable-dev+unsubscr...@lists.darktable.org >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> ____________________________________________________________ >>>> _______________ >>>> >> >> >> darktable developer mailing list >>>> >> >> >> to unsubscribe send a mail to >>>> >> >> >> darktable-dev+unsubscr...@lists.darktable.org >>>> >> >> >> >>>> >> >> > >>>> >> >> >>>> >> >> >>>> >> >> ____________________________________________________________ >>>> _______________ >>>> >> >> darktable developer mailing list >>>> >> >> to unsubscribe send a mail to >>>> >> >> darktable-dev+unsubscr...@lists.darktable.org >>>> >> >> >>>> >> > >>>> > >>>> > >>>> > ____________________________________________________________ >>>> _______________ >>>> > darktable developer mailing list to unsubscribe send a mail to >>>> > darktable-dev+unsubscr...@lists.darktable.org >>>> > >>>> > >>>> > >>>> > ____________________________________________________________ >>>> _______________ >>>> > darktable developer mailing list to unsubscribe send a mail to >>>> > darktable-dev+unsubscr...@lists.darktable.org >>>> ____________________________________________________________ >>>> _______________ >>>> darktable developer mailing list >>>> to unsubscribe send a mail to darktable-dev+unsubscribe@list >>>> s.darktable.org >>>> >>>> >>> ___________________________________________________________________________ >>> darktable developer mailing list to unsubscribe send a mail to >>> darktable-dev+unsubscr...@lists.darktable.org >>> >>> >>> >>> ___________________________________________________________________________ >>> darktable developer mailing list to unsubscribe send a mail to >>> darktable-dev+unsubscr...@lists.darktable.org >>> >> >> >> ___________________________________________________________________________ >> darktable developer mailing list to unsubscribe send a mail to >> darktable-dev+unsubscr...@lists.darktable.org >> >> >> > > > > > ___________________________________________________________________________ > darktable developer mailing list to unsubscribe send a mail to > darktable-dev+unsubscr...@lists.darktable.org > > > ___________________________________________________________________________ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
