No, Doug. I think I was the only one mis-interpreting the FDR steps, and I'm considerably closer to the Caret source than the Freesurfer source. ;-)
On 10/19/2009 11:18 AM, Douglas N Greve wrote: > I believe that the FreeSurfer version is also doing the right thing. > Do you have reason to believe otherwise? > > thanks > > doug > > Donna Dierker wrote: >> Mike Harms tried to explain that to me, but I was missing how the >> resampling affected the i term. Thanks, Tom, for spelling it out for >> me. ;-) >> >> Fortunately, John Harwell has less noise in his neural network than I >> do, so Caret does the right thing. (Just checked.) >> >> On 10/17/2009 10:29 AM, Doug Greve wrote: >> >>> Here's Tom Nichols explanation. He points out that you use the maximum >>> i, not the minimum, so you don't have to get past i=1. Thanks Tom! >>> >>> >>> ---------- Forwarded message ---------- >>> Date: Fri, 16 Oct 2009 23:11:09 +0100 >>> From: Thomas Nichols <t.e.nich...@warwick.ac.uk> >>> To: Doug Greve <gr...@nmr.mgh.harvard.edu> >>> Cc: Donna Dierker <do...@brainvis.wustl.edu> >>> Subject: Re: [Freesurfer] FDR correction (fwd) >>> >>> Dear Doug, >>> >>> Great discussion! But I think it misses the point. Can you forward >>> the >>> following to the list? >>> >>> The Benjamini-Hochberg (BH) FDR procedure uses the inequality >>> P(i) <= i / V * q >>> for the ordered P-values P(1)<=P(2)<=...<=P(V) and desired level-q FDR >>> control, which suggests that one will never get significance unless >>> P(1)<q/V, just like Bonferroni. HOWEVER, the key element of BH-FDR >>> is that >>> you get to search over i and find the *largest* P-value that >>> satisfies the >>> inequality and use that P-value as a threshold. >>> >>> Hence when considering changing resolutions (i.e. increasing V, but >>> increasing resolution/smoothness and roughly keeping the information >>> content >>> of the images the same) the magic of BH-FDR is that if the 5-th %ile of >>> uncorrected P-values is FDR-significant at one resolution, i.e. >>> i' = 0.05*V satisfies P( i' / V ) <= i' / V * q >>> then one would expect that the 5-th %ile of P-values after >>> up-sampling would >>> have a similar value, and thus would also satisfy the inequality >>> even though >>> V has changed. >>> >>> That is the motivation that FDR is resilient to changes in resolution. >>> However, I should note that in my own experience, leaving V fixed but >>> changing smoothness, often changes the distribution of P-values >>> dramatically, and thus changes the FDR result. But that should be a >>> different beast that up-interpolation. >>> >>> Hope this helps! >>> >>> -Tom >>> >>> On Fri, Oct 16, 2009 at 5:43 PM, Doug Greve >>> <gr...@nmr.mgh.harvard.edu>wrote: >>> >>> >>>> Tom, here's an FDR question for you. It appears that the FDR >>>> correction is dependent on the number of voxels (need p < fdr/N just >>>> to get past i=1). Meaning that as N grows, the min p-value must also >>>> shrink to get past i=1. Any way to get around this? >>>> >>>> thanks >>>> >>>> doug >>>> >>>> >>>> >>>> >>>> ---------- Forwarded message ---------- >>>> Date: Fri, 16 Oct 2009 11:30:59 -0500 >>>> From: Donna Dierker <do...@brainvis.wustl.edu> >>>> To: Michael Harms <mha...@conte.wustl.edu> >>>> Cc: freesurfer@nmr.mgh.harvard.edu, Yulia WORBE <yulia.wo...@upmc.fr> >>>> Subject: Re: [Freesurfer] FDR correction >>>> >>>> Regardless: FDR's sensitivity appears resolution-dependent to me. >>>> >>>> On 10/16/2009 10:39 AM, Michael Harms wrote: >>>> >>>> >>>>> Interesting post Donna, but my understanding of FDR is that it >>>>> sets the >>>>> p-value threshold based on the LARGEST p-value that satisfies the FDR >>>>> relationship. >>>>> >>>>> That is, steps 3 and 4 in Genovese et al. (2002) are: >>>>> 3) Let r be the largest i for which p <= i/V*q (assuming c=1) >>>>> 4) Threshold the image at the p-value p(r). >>>>> >>>>> So, it isn't the case that you require the most significant >>>>> p-value to >>>>> satisfy p <= 0.05/V "just to get past i=1" as you put it in your >>>>> post. >>>>> >>>>> Rather, you pick the largest p-value that satisfies the relationship, >>>>> meaning that lower (more-significant) p-values may not have >>>>> necessarily >>>>> satisfied p <= i/V*q for their particular position in the sorted >>>>> list of >>>>> p-values. >>>>> >>>>> cheers, >>>>> Mike H. >>>>> >>>>> >>>>> On Fri, 2009-10-16 at 10:13 -0500, Donna Dierker wrote: >>>>> >>>>> I never heard anything on my post here, but it might just be high >>>>> >>>>>> surface resolution: >>>>>> >>>>>> >>>>>> http://www.mail-archive.com/neuro-mult-c...@brainvis.wustl.edu/msg00026.html >>>>>> >>>>>> >>>>>> >>>>>> On 10/16/2009 09:58 AM, Michael Harms wrote: >>>>>> >>>>>> Your FDR analysis sounds correct. You probably have a rather small >>>>>> >>>>>>> number of "marginally" significant vertices, which is why none >>>>>>> survive >>>>>>> FDR. You could try increasing the "q" value from say 0.05 to >>>>>>> 0.1, in >>>>>>> which case 10% of the surviving vertices would be expected to be >>>>>>> false >>>>>>> positives. >>>>>>> >>>>>>> cheers, >>>>>>> Mike H. >>>>>>> >>>>>>> On Fri, 2009-10-16 at 12:03 +0200, Yulia WORBE wrote: >>>>>>> >>>>>>> >>>>>>> Dear Freesurfer team, >>>>>>> >>>>>>>> We are currently doing a cortical thickness studies between a >>>>>>>> group of >>>>>>>> psychiatric patients (n=60) and controls (n=30). We tested several >>>>>>>> smoothing levels (15mm, 20mm, 25mm) >>>>>>>> >>>>>>>> When setting an uncorrected threshold (such as p<0.005), we >>>>>>>> obtained >>>>>>>> several regions of decreased thickness, which are consistent >>>>>>>> with the >>>>>>>> pathology. >>>>>>>> >>>>>>>> However, when trying to correct for multiple comparisons using FDR >>>>>>>> ("Set Using FDR" button in qdec), the computed threshold is >>>>>>>> very high >>>>>>>> (e.g. 4.3 for 20mm smoothing) and, obviously, no significant >>>>>>>> regions >>>>>>>> are left. >>>>>>>> >>>>>>>> Did we do anything wrong in the analysis ? >>>>>>>> >>>>>>>> Thank you very much for your help, >>>>>>>> Yulia >>>>>>>> >>>>>>>> _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
