You're welcome ;-) That's true : the multiplication is equivalent to an "AND" operation, the resulting mask has non-zero values where both TV AND Laplacian masks has non-zero values, which - from my tests - is where the real noise is.
That is because TV alone is too sensitive : when the image is noisy, it works fine, but whenever the image is clean or barely noisy, it detect edges as well, thus false-positive in the case of noise detection. The TV × Laplacian is a safety jacket that allows the TV to work as expected on noisy images (see the example) but will protect sharp edges on clean images (on the example, the masks barely grabs a few pixels in the in-focus area). I have found that the only way we could overcome the oversensibility of the TV alone is by setting a window (like a band-pass filter) instead of a threshold (high-pass filter) because, in a noisy picture, depending on the noise level, the TV values of noisy and edgy pixels are very close. >From an end-user perspective, this is tricky. Using TV × Laplacian, given that the noise stats should not vary much for a given sensor at a given ISO, allows to confidently set a simple threshold as a factor of the standard deviation. It gives more reproductibility and allows to build preset/styles for given camera/ISO. Assuming gaussian noise, if you set your threshold factor to X (which means "unmask everything above the mean (TV × Laplacian) + X standard deviation), you know beforehand how many high-frequency pixels will be affected, no matter what : * X = -1 => 84 %, * 0 => 50 %, * 1 => 16 % , * 2 => 2.5 %, * 3 => 0.15 % * … Le 01/07/2018 à 14:13, rawfiner a écrit : > Thank you for this study Aurélien > > As far as I understand, TV and Laplacians are complementary as they > detect noise in different regions of the image (noise in sharp edge > for Laplacian, noise elsewhere for TV). > Though, I do not understand why you multiply the TV and Laplacian > results to get the mask. > Multiplying them would result in a mask containing non-zero values > only for pixels that are detected as noise both by TV and Laplacian. > Is there a particular reason for multiplying (or did I misunderstood > something?), or could we take the maximum value among TV and Laplacian > for each pixel instead? > > Thanks again > > Cheers, > rawfiner > > > 2018-07-01 3:45 GMT+02:00 Aurélien Pierre <rese...@aurelienpierre.com > <mailto:rese...@aurelienpierre.com>>: > > Hi, > > I have done experiments on that matter and took the opportunity to > correct/test further my code. > > So here are my attempts to code a noise mask and a sharpness mask > with total variation and laplacian norms : > > https://github.com/aurelienpierre/Image-Cases-Studies/blob/master/notebooks/Total%20Variation%20masking.ipynb > > <https://github.com/aurelienpierre/Image-Cases-Studies/blob/master/notebooks/Total%20Variation%20masking.ipynb> > > Performance benchmarks are at the end. > > Cheers, > > Aurélien. > > > Le 17/06/2018 à 15:03, rawfiner a écrit : >> >> >> Le dimanche 17 juin 2018, Aurélien Pierre >> <rese...@aurelienpierre.com <mailto:rese...@aurelienpierre.com>> >> a écrit : >> >> >> >> Le 13/06/2018 à 17:31, rawfiner a écrit : >>> >>> >>> Le mercredi 13 juin 2018, Aurélien Pierre >>> <rese...@aurelienpierre.com >>> <mailto:rese...@aurelienpierre.com>> a écrit : >>> >>> >>>> >>>> On Thu, Jun 14, 2018 at 12:23 AM, Aurélien Pierre >>>> <rese...@aurelienpierre.com >>>> <mailto:rese...@aurelienpierre.com>> wrote: >>>> > Hi, >>>> > >>>> > The problem of a 2-passes denoising method >>>> involving 2 differents >>>> > algorithms, the later applied where the former >>>> failed, could be the grain >>>> > structure (the shape of the noise) would be >>>> different along the picture, >>>> > thus very unpleasing. >>>> >>>> >>>> I agree that the grain structure could be different. >>>> Indeed, the grain could be different, but my feeling >>>> (that may be wrong) is that it would be still better >>>> than just no further processing, that leaves some >>>> pixels unprocessed (they could form grain structures >>>> far from uniform if we are not lucky). >>>> If you think it is only due to a change of algorithm, I >>>> guess we could apply non local means again on pixels >>>> where a first pass failed, but with different >>>> parameters to be quite confident that the second pass >>>> will work. >>> That sounds better to me… but practice will have the >>> last word. >>> >>> >>> Ok :-) >>> >>>> >>>> >>>> > >>>> > I thought maybe we could instead create some sort >>>> of total variation >>>> > threshold on other denoising modules : >>>> > >>>> > compute the total variation of each channel of >>>> each pixel as the divergence >>>> > divided by the L1 norm of the gradient - we then >>>> obtain a "heatmap" of the >>>> > gradients over the picture (contours and noise) >>>> > let the user define a total variation threshold >>>> and form a mask where the >>>> > weights above the threshold are the total >>>> variation and the weights below >>>> > the threshold are zeros (sort of a highpass >>>> filter actually) >>>> > apply the bilateral filter according to this mask. >>>> > >>>> > This way, if the user wants to stack several >>>> denoising modules, he could >>>> > protect the already-cleaned areas from further >>>> denoising. >>>> > >>>> > What do you think ? >>>> >>>> >>>> That sounds interesting. >>>> This would maybe allow to keep some small >>>> variations/details that are not due to noise or not >>>> disturbing, while denoising the other parts. >>>> Also, it may be computationally interesting (depends on >>>> the complexity of the total variation computation, I >>>> don't know it), as it could reduce the number of pixels >>>> to process. >>>> I guess the user could use something like that also the >>>> other way?: to protect high detailed zones and apply >>>> the denoising on quite smoothed zones only, in order to >>>> be able to use stronger denoising on zones that are >>>> supposed to be background blur. >>> >>> The noise is high frequency, so the TV (total variation) >>> threshold will have to be high pass only. The hypothesis >>> behind the TV thresholding is noisy pixels should have >>> abnormally higher gradients than true details, so you >>> isolate them this way. Selecting noise in low >>> frequencies areas would require in addition something >>> like a guided filter, which I believe is what is used in >>> the dehaze module. The complexity of the TV computation >>> depends on the order of accuracy you expect. >>> >>> A classic approximation of the gradient is using a >>> convolution product with Sobel or Prewitt operators (3×3 >>> arrays, very efficient, fairly accurate for edges, >>> probably less accurate for punctual noise). I have >>> developped myself optimized methods using 2, 4, and 8 >>> neighbouring pixels that give higher order accuracy, >>> given the sparsity of the data, at the expense of >>> computing cost : >>> >>> https://github.com/aurelienpierre/Image-Cases-Studies/blob/947fd8d5c2e4c3384c80c1045d86f8cf89ddcc7e/lib/deconvolution.pyx#L342 >>> >>> <https://github.com/aurelienpierre/Image-Cases-Studies/blob/947fd8d5c2e4c3384c80c1045d86f8cf89ddcc7e/lib/deconvolution.pyx#L342> >>> (ignore the variable ut in the code, only u is relevant >>> for us here). >>> >>> Great, thanks for the explanations. >>> Looking at the code of the 8 neighbouring pixels, I wonder >>> if we would make sense to compute something like that on raw >>> data considering only neighbouring pixels of the same color? >> >> the RAW data are even more sparse, so the gradient can't be >> computed this way. One would have to tweak the Taylor theorem >> to find an expression of gradient for sparse data. And that >> would be different for Bayer and X-Trans patterns. It's a bit >> of a conundrum. >> >> >> Ok, thank you for these explainations >> >> >>> >>> Also, when talking about the mask formed from the heat map, >>> do you mean that the "heat" would give for each pixel a >>> weight to use between input and output? (i.e. a mask that is >>> not only ones and zeros, but that controls how much input >>> and output are used for each pixel) >>> If so, I think it is a good idea to explore! >> yes, exactly, think of it as an opacity mask where you remap >> the user-input TV threshold and the lower values to 0, the >> max magnitude of TV to 1, and all the values in between >> accordingly. >> >> >> Ok that is really cool! It seems a good idea to try to use that! >> >> rawfiner >> >> >> >>> >>> rawfiner >>> >>>> >>>> >>>> >>>> > >>>> > Aurélien. >>>> > >>>> > >>>> > Le 13/06/2018 à 03:16, rawfiner a écrit : >>>> > >>>> > Hi, >>>> > >>>> > I don't have the feeling that increasing K is the >>>> best way to improve noise >>>> > reduction anymore. >>>> > I will upload the raw next week (if I don't >>>> forget to), as I am not at home >>>> > this week. >>>> > My feeling is that doing non local means on raw >>>> data gives much bigger >>>> > improvement than that. >>>> > I still have to work on it yet. >>>> > I am currently testing some raw downsizing ideas >>>> to allow a fast execution >>>> > of the algorithm. >>>> > >>>> > Apart of that, I also think that to improve noise >>>> reduction such as the >>>> > denoise profile in nlm mode and the denoise non >>>> local means, we could do a 2 >>>> > passes algorithm, with non local means applied >>>> first, and then a bilateral >>>> > filter (or median filter or something else) >>>> applied only on pixels where non >>>> > local means failed to find suitable patches (i.e. >>>> pixels where the sum of >>>> > weights was close to 0). >>>> > The user would have a slider to adjust this setting. >>>> > I think that it would make easier to have a >>>> "uniform" output (i.e. an output >>>> > where noise has been reduced quite uniformly) >>>> > I have not tested this idea yet. >>>> > >>>> > Cheers, >>>> > rawfiner >>>> > >>>> > Le lundi 11 juin 2018, johannes hanika >>>> <hana...@gmail.com <mailto:hana...@gmail.com>> a >>>> écrit : >>>> >> >>>> >> hi, >>>> >> >>>> >> i was playing with noise reduction presets again >>>> and tried the large >>>> >> neighbourhood search window. on my shots i could >>>> very rarely spot a >>>> >> difference at all increasing K above 7, and even >>>> less so going above >>>> >> 10. the image you posted earlier did show quite >>>> a substantial >>>> >> improvement however. i was wondering whether >>>> you'd be able to share >>>> >> the image so i can evaluate on it? maybe i just >>>> haven't found the >>>> >> right test image yet, or maybe it's camera >>>> dependent? >>>> >> >>>> >> (and yes, automatic and adaptive would be better >>>> but if we can ship a >>>> >> simple slider that can improve matters, maybe we >>>> should) >>>> >> >>>> >> cheers, >>>> >> jo >>>> >> >>>> >> >>>> >> >>>> >> On Mon, Jan 29, 2018 at 2:05 AM, rawfiner >>>> <rawfi...@gmail.com <mailto:rawfi...@gmail.com>> wrote: >>>> >> > Hi >>>> >> > >>>> >> > Yes, the patch size is set to 1 from the GUI, >>>> so it is not a bilateral >>>> >> > filter, and I guess it corresponds to a patch >>>> window size of 3x3 in the >>>> >> > code. >>>> >> > The runtime difference is near the expected >>>> quadratic slowdown: >>>> >> > 1,460 secs (8,379 CPU) for 7 and 12,794 secs >>>> (85,972 CPU) for 25, which >>>> >> > means about 10.26x slowdown >>>> >> > >>>> >> > If you want to make your mind on it, I have >>>> pushed a branch here that >>>> >> > integrates the K parameter in the GUI: >>>> >> > https://github.com/rawfiner/darktable.git >>>> <https://github.com/rawfiner/darktable.git> >>>> >> > The branch is denoise-profile-GUI-K >>>> >> > >>>> >> > I think that it may be worth to see if an >>>> automated approach for the >>>> >> > choice >>>> >> > of K may work, in order not to integrate the >>>> parameter in the GUI. >>>> >> > I may try to implement the approach of Kervann >>>> and Boulanger (the >>>> >> > reference >>>> >> > from the darktable blog post) to see how it >>>> performs. >>>> >> > >>>> >> > cheers, >>>> >> > rawfiner >>>> >> > >>>> >> > >>>> >> > 2018-01-27 13:50 GMT+01:00 johannes hanika >>>> <hana...@gmail.com <mailto:hana...@gmail.com>>: >>>> >> >> >>>> >> >> heya, >>>> >> >> >>>> >> >> thanks for the reference! interesting >>>> interpretation how the blotches >>>> >> >> form. not sure i'm entirely convinced by that >>>> argument. >>>> >> >> your image does look convincing though. let >>>> me get this right.. you >>>> >> >> ran with radius 1 which means patch window >>>> size 3x3? not 1x1 which >>>> >> >> would be a bilateral filter effectively? >>>> >> >> >>>> >> >> also what was the run time difference? is it >>>> near the expected >>>> >> >> quadratic slowdown from 7 (i.e. 15x15) to 25 >>>> (51x51) so about 11.56x >>>> >> >> slower with the large window size? (test with >>>> darktable -d perf) >>>> >> >> >>>> >> >> since nlmeans isn't the fastest thing, even >>>> with this coalesced way of >>>> >> >> implementing it, we should certainly keep an >>>> eye on this. >>>> >> >> >>>> >> >> that being said if we can often times get >>>> much better results we >>>> >> >> should totally expose this in the gui, maybe >>>> with a big warning that >>>> >> >> it really severely impacts speed. >>>> >> >> >>>> >> >> cheers, >>>> >> >> jo >>>> >> >> >>>> >> >> On Sat, Jan 27, 2018 at 7:34 AM, rawfiner >>>> <rawfi...@gmail.com <mailto:rawfi...@gmail.com>> wrote: >>>> >> >> > Thank you for your answer >>>> >> >> > I perfectly agree with the fact that the >>>> GUI should not become >>>> >> >> > overcomplicated. >>>> >> >> > >>>> >> >> > As far as I understand, the pixels within a >>>> small zone may suffer >>>> >> >> > from >>>> >> >> > correlated noise, and there is a risk of >>>> noise to noise matching. >>>> >> >> > That's why this paper suggest not to take >>>> pixels that are too close >>>> >> >> > to >>>> >> >> > the >>>> >> >> > zone we are correcting, but to take them a >>>> little farther (see the >>>> >> >> > caption >>>> >> >> > of Figure 2 for a quick explaination): >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >>>> https://pdfs.semanticscholar.org/c458/71830cf535ebe6c2b7656f6a205033761fc0.pdf >>>> >>>> <https://pdfs.semanticscholar.org/c458/71830cf535ebe6c2b7656f6a205033761fc0.pdf> >>>> >> >> > (in case you ask, unfortunately there is a >>>> patent associated with >>>> >> >> > this >>>> >> >> > approach, so we cannot implement it) >>>> >> >> > >>>> >> >> > Increasing the neighborhood parameter >>>> results in having >>>> >> >> > proportionally >>>> >> >> > less >>>> >> >> > problem of correlation between surrounding >>>> pixels, and decreases the >>>> >> >> > size of >>>> >> >> > the visible spots. >>>> >> >> > See for example the two attached pictures: >>>> one with size 1, force 1, >>>> >> >> > and >>>> >> >> > K 7 >>>> >> >> > and the other with size 1, force 1, and K 25. >>>> >> >> > >>>> >> >> > I think that the best would probably be to >>>> adapt K automatically, in >>>> >> >> > order >>>> >> >> > not to affect the GUI, and as we may have >>>> different levels of noise >>>> >> >> > in >>>> >> >> > different parts of an image. >>>> >> >> > In this post >>>> >> >> > >>>> >> >> > >>>> >>>> (https://www.darktable.org/2012/12/profiling-sensor-and-photon-noise/ >>>> >>>> <https://www.darktable.org/2012/12/profiling-sensor-and-photon-noise/>), >>>> >> >> > this >>>> >> >> > paper is cited: >>>> >> >> > >>>> >> >> > [4] charles kervrann and jerome boulanger: >>>> optimal spatial adaptation >>>> >> >> > for >>>> >> >> > patch-based image denoising. ieee trans. >>>> image process. vol. 15, no. >>>> >> >> > 10, >>>> >> >> > 2006 >>>> >> >> > >>>> >> >> > As far as I understand, it gives a way to >>>> choose an adaptated window >>>> >> >> > size >>>> >> >> > for each pixel, but I don't see in the code >>>> anything related to that >>>> >> >> > >>>> >> >> > Maybe is this paper related to the TODOs in >>>> the code ? >>>> >> >> > >>>> >> >> > Was it planned to implement such a variable >>>> window approach ? >>>> >> >> > >>>> >> >> > Or if it is already implemented, could you >>>> point me where ? >>>> >> >> > >>>> >> >> > Thank you >>>> >> >> > >>>> >> >> > rawfiner >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> > 2018-01-26 9:05 GMT+01:00 johannes hanika >>>> <hana...@gmail.com <mailto:hana...@gmail.com>>: >>>> >> >> >> >>>> >> >> >> hi, >>>> >> >> >> >>>> >> >> >> if you want, absolutely do play around >>>> with K. in my tests it did >>>> >> >> >> not >>>> >> >> >> lead to any better denoising. to my >>>> surprise a larger K often led to >>>> >> >> >> worse results (for some reason often the >>>> relevance of discovered >>>> >> >> >> patches decreases with distance from the >>>> current point). that's why >>>> >> >> >> K >>>> >> >> >> is not exposed in the gui, no need for >>>> another irrelevant and >>>> >> >> >> cryptic >>>> >> >> >> parameter. if you find a compelling case >>>> where this indeed leads to >>>> >> >> >> better denoising we could rethink that. >>>> >> >> >> >>>> >> >> >> in general NLM is a 0-th order denoising >>>> scheme, meaning the prior >>>> >> >> >> is >>>> >> >> >> piecewise constant (you claim the pixels >>>> you find are trying to >>>> >> >> >> express /the same/ mean, so you average >>>> them). if you let that >>>> >> >> >> algorithm do what it would really like to, >>>> it'll create unpleasant >>>> >> >> >> blotches of constant areas. so for best >>>> results we need to tone it >>>> >> >> >> down one way or another. >>>> >> >> >> >>>> >> >> >> cheers, >>>> >> >> >> jo >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> On Fri, Jan 26, 2018 at 7:36 AM, rawfiner >>>> <rawfi...@gmail.com <mailto:rawfi...@gmail.com>> >>>> >> >> >> wrote: >>>> >> >> >> > Hi >>>> >> >> >> > >>>> >> >> >> > I am surprised to see that we cannot >>>> control the neighborhood >>>> >> >> >> > parameter >>>> >> >> >> > for >>>> >> >> >> > the NLM algorithm (neither for the >>>> denoise non local mean, nor for >>>> >> >> >> > the >>>> >> >> >> > denoise profiled) from the GUI. >>>> >> >> >> > I see in the code (denoiseprofile.c) >>>> this TODO that I don't >>>> >> >> >> > understand: >>>> >> >> >> > "// >>>> >> >> >> > TODO: fixed K to use adaptive size >>>> trading variance and bias!" >>>> >> >> >> > And just some lines after that: "// >>>> TODO: adaptive K tests here!" >>>> >> >> >> > (K is the neighborhood parameter of the >>>> NLM algorithm). >>>> >> >> >> > >>>> >> >> >> > In practice, I think that being able to >>>> change the neighborhood >>>> >> >> >> > parameter >>>> >> >> >> > allows to have a better noise reduction >>>> for one image. >>>> >> >> >> > For example, choosing a bigger K allows >>>> to reduce the spotted >>>> >> >> >> > aspect >>>> >> >> >> > that >>>> >> >> >> > one can get on high ISO images. >>>> >> >> >> > >>>> >> >> >> > Of course, increasing K increase >>>> computational time, but I think >>>> >> >> >> > we >>>> >> >> >> > could >>>> >> >> >> > find an acceptable range that would >>>> still be useful. >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> > Is there any reason for not letting the >>>> user control the >>>> >> >> >> > neighborhood >>>> >> >> >> > parameter in the GUI ? >>>> >> >> >> > Also, do you understand the TODOs ? >>>> >> >> >> > I feel that we would probably get better >>>> denoising by fixing >>>> >> >> >> > these, >>>> >> >> >> > but >>>> >> >> >> > I >>>> >> >> >> > don't understand them. >>>> >> >> >> > >>>> >> >> >> > I can spend some time on these TODOs, or >>>> to add the K parameter to >>>> >> >> >> > the >>>> >> >> >> > interface if you think it is worth it (I >>>> think so but it is only >>>> >> >> >> > my >>>> >> >> >> > personal >>>> >> >> >> > opinion), but I have to understand what >>>> the TODOs mean before >>>> >> >> >> > >>>> >> >> >> > Thank you for your help >>>> >> >> >> > >>>> >> >> >> > rawfiner >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> > >>>> >>>> ___________________________________________________________________________ >>>> >> >> >> > darktable developer mailing list to >>>> unsubscribe send a mail to >>>> >> >> >> > >>>> darktable-dev+unsubscr...@lists.darktable.org >>>> <mailto:darktable-dev+unsubscr...@lists.darktable.org> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >>>> ___________________________________________________________________________ >>>> >> >> >> darktable developer mailing list >>>> >> >> >> to unsubscribe send a mail to >>>> >> >> >> >>>> darktable-dev+unsubscr...@lists.darktable.org >>>> <mailto:darktable-dev+unsubscr...@lists.darktable.org> >>>> >> >> >> >>>> >> >> > >>>> >> >> >>>> >> >> >>>> >> >> >>>> >>>> ___________________________________________________________________________ >>>> >> >> darktable developer mailing list >>>> >> >> to unsubscribe send a mail to >>>> >> >> darktable-dev+unsubscr...@lists.darktable.org >>>> <mailto:darktable-dev+unsubscr...@lists.darktable.org> >>>> >> >> >>>> >> > >>>> > >>>> > >>>> > >>>> >>>> ___________________________________________________________________________ >>>> > darktable developer mailing list to unsubscribe >>>> send a mail to >>>> > darktable-dev+unsubscr...@lists.darktable.org >>>> <mailto:darktable-dev+unsubscr...@lists.darktable.org> >>>> > >>>> > >>>> > >>>> > >>>> >>>> ___________________________________________________________________________ >>>> > darktable developer mailing list to unsubscribe >>>> send a mail to >>>> > darktable-dev+unsubscr...@lists.darktable.org >>>> <mailto:darktable-dev+unsubscr...@lists.darktable.org> >>>> >>>> ___________________________________________________________________________ >>>> darktable developer mailing list >>>> to unsubscribe send a mail to >>>> darktable-dev+unsubscr...@lists.darktable.org >>>> <mailto:darktable-dev+unsubscr...@lists.darktable.org> >>>> >>>> >>>> >>>> ___________________________________________________________________________ >>>> darktable developer mailing list to unsubscribe send a >>>> mail to darktable-dev+unsubscr...@lists.darktable.org >>>> <mailto:darktable-dev+unsubscr...@lists.darktable.org> >>> >>> >>> >>> ___________________________________________________________________________ >>> darktable developer mailing list to unsubscribe send a >>> mail to darktable-dev+unsubscr...@lists.darktable.org >>> <mailto:darktable-dev%2bunsubscr...@lists.darktable.org> >>> >>> >>> >>> >>> ___________________________________________________________________________ >>> darktable developer mailing list to unsubscribe send a mail >>> to darktable-dev+unsubscr...@lists.darktable.org >>> <mailto:darktable-dev+unsubscr...@lists.darktable.org> >> >> >> >> >> >> >> >> ___________________________________________________________________________ >> darktable developer mailing list to unsubscribe send a mail to >> darktable-dev+unsubscr...@lists.darktable.org >> <mailto:darktable-dev+unsubscr...@lists.darktable.org> > > ___________________________________________________________________________ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
