By "wrong" I meant that permutation no longer gives exact p-values in expectation with non-orthogonal designs. There is no theory to characterize the accuracy of Freeman-Lane of the other methods. In this sense, they are not approximations but ad hoc methods that people hope do a better job than parametric methods. Anderson tested them on a wide range of designs, but it was, of course, not exhaustive. The accuracy of his results may not extend to other designs, so it is a buyer-beware situation (as with all neuroimaging).
On 8/29/16 10:16 PM, Harms, Michael wrote: > Hi, > I wouldn’t say that non-orthogonal designs are “wrong” to use with > permutation. Rather, there are different approaches to handling that > situation and produce approximate p-values. See Table 2 in Winkler’s 2014 > paper, and the results therein comparing the various approaches: > > http://www.ncbi.nlm.nih.gov/pubmed/24530839 > > > PALM actually gives you control over the method used, with the default > (and recommended) approach being that of “Freedman-Lane", which is the > same approach used by FSL’s ‘randomise’ tool to handle correlated > covariates. > > cheers, > -MH > > -- > Michael Harms, Ph.D. > > ----------------------------------------------------------- > Conte Center for the Neuroscience of Mental Disorders > Washington University School of Medicine > Department of Psychiatry, Box 8134 > 660 South Euclid Ave.Tel: 314-747-6173 > St. Louis, MO 63110Email: mha...@wustl.edu > > > > > On 8/29/16, 7:49 PM, "freesurfer-boun...@nmr.mgh.harvard.edu on behalf of > Matt Glasser" <freesurfer-boun...@nmr.mgh.harvard.edu on behalf of > m...@ma-tea.com> wrote: > > PALM handles GIFTI and CIFTI data. > > Peace, > > Matt. > > On 8/29/16, 6:21 PM, "Douglas N Greve" > <freesurfer-boun...@nmr.mgh.harvard.edu on behalf of > gr...@nmr.mgh.harvard.edu> wrote: > >> Does PALM do surface-based? Also, there is no way to appropriately >> handle this. For permutation, non-orthogonal designs are wrong. There >> are ways to try to compensate for it, which is what PALM is doing. Sorry >> to be nit-picky! >> >> >> On 08/29/2016 06:12 PM, Harms, Michael wrote: >>> Hi Maaike, >>> Why not just use PALM? Then you don¹t have to worry about this (since >>> PALM appropriately handles the situation of correlated covariates). >>> >>> cheers, >>> -MH >>> >>> -- >>> Michael Harms, Ph.D. >>> >>> ----------------------------------------------------------- >>> Conte Center for the Neuroscience of Mental Disorders >>> Washington University School of Medicine >>> Department of Psychiatry, Box 8134 >>> 660 South Euclid Ave.Tel: 314-747-6173 >>> St. Louis, MO 63110Email: mha...@wustl.edu >>> >>> >>> >>> >>> On 8/29/16, 4:45 PM, "freesurfer-boun...@nmr.mgh.harvard.edu on behalf >>> of >>> Douglas N Greve" <freesurfer-boun...@nmr.mgh.harvard.edu on behalf of >>> gr...@nmr.mgh.harvard.edu> wrote: >>> >>> It is hard to say. Since the subjects are not exchangeable, the >>> permutation is technically not appropriate. Check the winkler paper, I >>> think he talks about what happens if you just don't do anything. >>> >>> >>> On 08/29/2016 11:07 AM, maaike rive wrote: >>>> Hi all, >>>> >>>> >>>> Is using forced permutation for non-orthogonal design matrices wrong >>>> or is it allowed to do this instead of using tools like palm (what >>>> happens eg with the covariates when using forced permutation)? I >>>> used forced permutation and it seemed to work, results were (partly) >>>> comparable to what I found with monte carlo simulations. >>>> >>>> >>>> Thanks, Maaike >>>> >>>> >>>> ------------------------------------------------------------------------ >>>> *Van:* freesurfer-boun...@nmr.mgh.harvard.edu >>>> <freesurfer-boun...@nmr.mgh.harvard.edu> namens Harms, Michael >>>> <mha...@wustl.edu> >>>> *Verzonden:* vrijdag 26 augustus 2016 01:00:13 >>>> *Aan:* Freesurfer support list >>>> *Onderwerp:* Re: [Freesurfer] mri_glmfit-sim permutation testing >>>> running after 3 days! >>>> >>>> Hi, >>>> You might want to check out FSL¹s PALM tool, which has a bit more >>>> sophisticated permutation framework, and allows for permutation in the >>>> context of non-orthogonal covariates. >>>> >>>> cheers, >>>> -MH >>>> >>>> -- >>>> Michael Harms, Ph.D. >>>> ----------------------------------------------------------- >>>> Conte Center for the Neuroscience of Mental Disorders >>>> Washington University School of Medicine >>>> Department of Psychiatry, Box 8134 >>>> 660 South Euclid Ave.Tel: 314-747-6173 >>>> St. Louis, MO 63110Email: mha...@wustl.edu >>>> >>>> From: <freesurfer-boun...@nmr.mgh.harvard.edu >>>> <mailto:freesurfer-boun...@nmr.mgh.harvard.edu>> on behalf of Ajay >>>> Kurani <dr.ajay.kur...@gmail.com <mailto:dr.ajay.kur...@gmail.com>> >>>> Reply-To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edu >>>> <mailto:freesurfer@nmr.mgh.harvard.edu>> >>>> Date: Thursday, August 25, 2016 at 4:13 PM >>>> To: Freesurfer support list <freesurfer@nmr.mgh.harvard.edu >>>> <mailto:freesurfer@nmr.mgh.harvard.edu>> >>>> Subject: Re: [Freesurfer] mri_glmfit-sim permutation testing running >>>> after 3 days! >>>> >>>> Hi Doug, >>>> Thanks for the help! I think I figured out the issue based on your >>>> response. >>>> 1) I created a template to use for this group and named it fsaverage >>>> (including creating monte carlo simulations) for simplicity of >>>> integrating with freesurfer as I am newer to it. This is why the >>>> sizes didn't match up as you expected >>>> but the mri_glmfit still ran. >>>> >>>> 2) I deleted the folder and restarted without background processes. >>>> The error became apparent. Of my covariates (2 fix factors and 3 >>>> quantitative), not all were orthogonal. In looking at the error more, >>>> it seems that i need to add the >>>> --perm-force if I wanted the simulation to run, however the background >>>> processes were not aware of this error and kept polling as you >>>> mentioned. >>>> >>>> This brings me to a new but related issue. From what I have read in >>>> other freesurfer posts, it is statistically incorrect to use >>>> --perm-force for non-orthogonal covariates (or continuous covariates). >>>> I am unsure how to proceed. >>>> a) If I ran permutation testing (to overcome the issue of incorrect >>>> smoothness estimations from the gaussian distribution assumption), >>>> then I run into the issue of non-orthogonal covariates. Is there a >>>> way to orthogonalize the data in >>>> freesurfer, or a solution to this issue? >>>> >>>> b) If orthogonalizing is difficult to implement, another option is >>>> running Qdec with the montecarlo simulation at a more conservative p >>>> value (p< 0.001). From your previous posts, the testing at this p >>>> value for 10mm seems to meet the 5% FPR. One question is if the >>>> non-orthogonal data affects this analysis as well for this model? >>>> >>>> Thanks, >>>> Ajay >>>> >>>> On Thu, Aug 25, 2016 at 12:18 PM, Ajay Kurani >>>> <dr.ajay.kur...@gmail.com <mailto:dr.ajay.kur...@gmail.com>> wrote: >>>> >>>> Hi Freesurfer Experts, >>>> I am trying to use freesurfer's mri_glmfit-sim tool to run >>>> permutation testing on cortical thickness data (as recommended by >>>> Doug in my previous post: >>>> >>>> >>>> http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.ht >>>> ml >>>> >>>> >>>> <http://www.mail-archive.com/freesurfer%40nmr.mgh.harvard.edu/msg48653.h >>>> tm >>>> l> >>>> ) >>>> >>>> Most of the tutorials I found were not related to permutation >>>> testing so the subsequent steps may be incorrect. Please let me >>>> know where I go wrong... >>>> >>>> 1) I first ran QDec to generate a folder for the analysis which >>>> would create the subsequent fsgd and y files needed my >>>> mri_glmfit-sim. I am running both left and right hemisphere >>>> cortical thickness analysis with 10mm smoothing. The following is >>>> for just the left hemisphere. Note I am doing a 3 group >>>> comparison, but for this 2 group ttest I manually centered the >>>> data based on the 3 group mean for age and education. >>>> >>>> 2) I ran the following command: >>>> /mri_glmfit-sim --glmdir ./HCvsPAT_lh_thickness_10mm/ --sim perm >>>> 10000 2 perm.abs.2 --sim-sign abs --bg 16 >>>> >>>> Prior to running the command above, from the y.fsdg file I deleted >>>> the fwhm estimate of 13mm since this was not correctly estimated >>>> (ACF with long tails). I assumed that by removing this estimate, >>>> it would force the permutation test to calculate based on the data >>>> but when looking at the log output I see the following which says >>>> fwhm 0: >>>> >>>> cmdline mri_glmfit.bin --C >>>> >>>> >>>> ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Avg-Intercept-t >>>> hi >>>> ckness.mtx >>>> --C >>>> >>>> >>>> ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-Male-Femal >>>> e- >>>> Intercept-thickness.mtx >>>> --C >>>> >>>> >>>> ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-Diff-PD-MCI-Int >>>> er >>>> cept-thickness.mtx >>>> --C >>>> >>>> >>>> ./HCvsPAT_lh_thickness_10mm//tmp.mri_glmfit-sim-19468/lh-X-Gender-Group- >>>> In >>>> tercept-thickness.mtx >>>> --sim perm 625 2 ./HCvsPAT_lh_thickness_10mm//csd/perm.abs.2.j013 >>>> --y >>>> >>>> >>>> /home/akurani/Documents/PPMI/FS_Final/qdec/HCvsPAT_lh_thickness_10mm/y.m >>>> gh >>>> --mask ./HCvsPAT_lh_thickness_10mm//mask.mgh --sim-sign abs --fwhm >>>> 0 --fsgd ./HCvsPAT_lh_thickness_10mm//y.fsgd dods --surf fsaverage >>>> lh white --sim-done >>>> ./HCvsPAT_lh_thickness_10mm//csd/poll/done.perm.abs.2.j013 >>>> >>>> 3)I started this a few days ago on a 16 core machine and it is >>>> still running in the terminal. I have 150 subjects in the >>>> analysis and specified 10000 iterations. In the terminal I >>>> assumed when I reach Poll 10000 it would be complete but currently >>>> I am at : Poll 13341 job 1 Thu Aug 25 12:03:51 CDT 2016 >>>> >>>> Questions: >>>> a) I am curious, is this going to run 10,000 simulations X 150 >>>> patients or does the Poll number not have anything to do with the >>>> number of iterations it is on? >>>> >>>> >>>> b) Did I run this procedure correctly? Was I incorrect in deleting >>>> the fwhm estimate from y.fsgd file generated by Qdec even though >>>> we know the estimate is incorrect since smoothness assumed a >>>> gaussian distribution as opposed to gaussian with heavy tails >>>> >>>> c) I noticed in the logfile the following warning: >>>> INFO: gd2mtx_method is dods >>>> Computing normalized matrix >>>> Normalized matrix condition is 5.65727 >>>> Matrix condition is 935.597 >>>> Found 136777 voxels in mask >>>> Reshaping mriglm->mask... >>>> search space = 89675.729228 >>>> ERROR: design matrix is not orthogonal, cannot be used with >>>> permutation. >>>> If this something you really want to do, run with --perm-force >>>> Poll 2 job 1 Tue Aug 23 22:58:18 CDT 2016 >>>> >>>> I am not sure if this means my simulation is incorrect? >>>> >>>> Thanks, >>>> Ajay >>>> >>>> >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> The materials in this message are private and may contain Protected >>>> Healthcare Information or other information of a sensitive nature. If >>>> you are not the intended recipient, be advised that any unauthorized >>>> use, disclosure, copying or the taking of any action in reliance on >>>> the contents of this information is strictly prohibited. If you have >>>> received this email in error, please immediately notify the sender via >>>> telephone or return mail. >>>> >>>> >>>> >>>> _______________________________________________ >>>> Freesurfer mailing list >>>> Freesurfer@nmr.mgh.harvard.edu >>>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer >>> -- >>> Douglas N. Greve, Ph.D. >>> MGH-NMR Center >>> gr...@nmr.mgh.harvard.edu >>> Phone Number: 617-724-2358 >>> Fax: 617-726-7422 >>> >>> Bugs: surfer.nmr.mgh.harvard.edu/fswiki/BugReporting >>> FileDrop:https://gate.nmr.mgh.harvard.edu/filedrop2 >>> www.nmr.mgh.harvard.edu/facility/filedrop/index.html >>> Outgoing:ftp://surfer.nmr.mgh.harvard.edu/transfer/outgoing/flat/greve/ >>> >>> _______________________________________________ >>> Freesurfer mailing list >>> Freesurfer@nmr.mgh.harvard.edu >>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer >>> >>> >>> The information in this e-mail is intended only for the person to whom >>> it >>> is >>> addressed. If you believe this e-mail was sent to you in error and the >>> e-mail >>> contains patient information, please contact the Partners Compliance >>> HelpLine at >>> http://www.partners.org/complianceline . If the e-mail was sent to you >>> in >>> error >>> but does not contain patient information, please contact the sender and >>> properly >>> dispose of the e-mail. >>> >>> >>> >>> ________________________________ >>> The materials in this message are private and may contain Protected >>> Healthcare Information or other information of a sensitive nature. If >>> you are not the intended recipient, be advised that any unauthorized >>> use, disclosure, copying or the taking of any action in reliance on the >>> contents of this information is strictly prohibited. If you have >>> received this email in error, please immediately notify the sender via >>> telephone or return mail. >>> >>> _______________________________________________ >>> Freesurfer mailing list >>> Freesurfer@nmr.mgh.harvard.edu >>> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer >>> >>> >> -- >> Douglas N. Greve, Ph.D. >> MGH-NMR Center >> gr...@nmr.mgh.harvard.edu >> Phone Number: 617-724-2358 >> Fax: 617-726-7422 >> >> Bugs: surfer.nmr.mgh.harvard.edu/fswiki/BugReporting >> FileDrop: https://gate.nmr.mgh.harvard.edu/filedrop2 >> www.nmr.mgh.harvard.edu/facility/filedrop/index.html >> Outgoing: ftp://surfer.nmr.mgh.harvard.edu/transfer/outgoing/flat/greve/ >> >> _______________________________________________ >> Freesurfer mailing list >> Freesurfer@nmr.mgh.harvard.edu >> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > > _______________________________________________ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > > ________________________________ > The materials in this message are private and may contain Protected > Healthcare Information or other information of a sensitive nature. If you are > not the intended recipient, be advised that any unauthorized use, disclosure, > copying or the taking of any action in reliance on the contents of this > information is strictly prohibited. If you have received this email in error, > please immediately notify the sender via telephone or return mail. > > _______________________________________________ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
