Dear Anderson, Thank you for that very thoughtful reply! I think your argument is convicing and I've decided to use FDR for each comparison individually.
Lars On Thu, Mar 26, 2009 at 5:37 PM, Anderson Winkler <relk...@bol.com.br>wrote: > Dear Lars, Pedro and all, > > There should be no problem in using different thresholds for each of these > pairwise comparisons. > > Let's consider the following extreme case: Suppose in one of these > comparisons you have a strong effect (say, "activation") that comprises > large brain regions. Suppose that for the 5 other comparisons, there is no > effect at all. If you pool all your six comparisons to calculate a single > threshold for all of them, due to the adaptiveness of the Benjamini & > Hochberg procedure, the calculated threshold will be such that it will > produce more liberal results (i.e. higher p-threshold) for the 5 comparisons > where there is no experimental effect, than would be obtained by not > pooling, implying that some vertices where the null is true will be > (falsely) declared as positive for these comparisons. > On the other hand, the vertices where there is no effect will also give > their contribution to compute this threshold, but their influence will be in > the opposite direction, producing more conservative results (i.e. lower > p-threshold) for the comparison where "activation" is present, than would be > without pooling, resulting in "active" vertices remaining not detected > (false negatives). > > Both are clearly undesirable, as the amount of errors (both type I and II) > is increased by bleeding the effect/absence of effect from one comparison > into another. > > In other words, when including in the same analysis different sets of > comparisons (which possibly includes different experimental hypotheses, > which definitely would preclude pooling), one will be losing one of the > nicest features of the B&H procedure: the weak control of FWE (i.e. when the > null is true everywhere, you are controlling FWE, even using an FDR > procedure) for each comparison if the null for any of these comparisons is > true everywhere. > > This also means that, although FDR would still be controlled globally, one > cannot make inferences about each comparison individually, which I believe > was the whole point of making the comparisons initially. > > Hope this helps! > > Kind regards, > > Anderson > > > > Lars M. Rimol wrote: > > Well, I believe there is a problem in principle here. FDR deals with > multiple comparisons across the surface (or brain volume), but how do you > deal with a series of such analyses? Of course, if you use a different > method of correction you avoid this problem but that's not the point. > > > LMR > > > ------------------------------ > Date: Tue, 24 Mar 2009 14:03:46 -0300 > Subject: Re: [Freesurfer] FDR correction > From: p...@netfilter.com.br > To: lari...@gmail.com > CC: freesurfer@nmr.mgh.harvard.edu > > Some links that may be helpful: > > http://surfer.nmr.mgh.harvard.edu/fswiki/FsTutorial/QdecMultipleComparisons > http://surfer.nmr.mgh.harvard.edu/fswiki/FsTutorial/GroupAnalysis > http://surfer.nmr.mgh.harvard.edu/fswiki/MultipleComparisons > > Hope it helps. > > PPJ > ----------------------------------------------------------- > Pedro Paulo de M. Oliveira Junior > Diretor de Operações > Netfilter & SpeedComm Telecom > > > > On Tue, Mar 24, 2009 at 12:38, Lars M. Rimol <lari...@gmail.com> wrote: > > Hi, > I have done an analysis involving three groups, so there are three pairwise > comparisons across two hemispheres = 6 p-maps. I want to adjust for multiple > comparisons (across the vertices), so I use FDR. But since FDR determines > the threshold basd on the actual p-values, I get 6 different tresholds: > > comparison 1: lh and rh, 0.016 and 0.028 (I can choose .01) > comparison 2: lh and rh, 0.01 and 0.001 (I can choose.001) > comparison 3 lh and rh, 0.001 and 0.0001 (I can choose .0001) > > There are lots of significant vertices in comparison 1 and nothing > significant, after correction, in comparison 3. Is there anything wrong with > using different tresholds here, and concluding that in comparison 1 there > were extensive differences between the groups, whereas in comparison 3 there > were none? I'm not sure if this is a problem, but I'm afraid some reviewers > might have an issue with it. Across the hemispheres, I can choose a > conservative threshold which covers both hemispheres, i.e. lower than both > the FDR-adjusted treshold for lh and rh. But between the comparisons the > tresholds differ even more, by a factor of 10 and 100. And if I choose the > most conservative of all the adjusted thresholds, I'm afraid that I'll make > a type II error in comparison 1. > > From what I understand, the adjusted threshold for comparison 3 is more > conservative because of the actual empirical data (the distribution of > p-values), so that's an empirical argument for using a more conservative > threshold there. > > And: What if I pooled all thre p-maps (sig.mgh) and did an FDR on the whole > thing, would that be a better approach? And does Freesurfer use the > Benjamini algorithm, and if you do, can I use Tom Nichols' matlab function > for FDR > (http://www.sph.umich.edu/~nichols/FDR/FDR.m<http://www.sph.umich.edu/%7Enichols/FDR/FDR.m>) > for pooling all three p-maps? > Thank you! > > -- > yours, > LMR > > > _______________________________________________ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > > ------------------------------ > > _______________________________________________ > Freesurfer mailing > listfreesur...@nmr.mgh.harvard.eduhttps://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > -- yours, LMR
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
