Thanks very much for getting back so quickly. The processing times were 5-6 hours for all instances. There is an incremental but only modest improvement for 4/6 vs 8/12 processors. I'll put up more information on the list when I've finished thinking this through.
Regards, Don Don Krieger, Ph.D. Department of Neurological Surgery University of Pittsburgh (412)648-9654 Office (412)521-4431 Cell/Text > -----Original Message----- > From: freesurfer-boun...@nmr.mgh.harvard.edu [mailto:freesurfer- > boun...@nmr.mgh.harvard.edu] On Behalf Of zkauf...@nmr.mgh.harvard.edu > Sent: Friday, May 01, 2015 1:41 AM > To: Freesurfer support list > Subject: Re: [Freesurfer] -openmp 8 and -use-gpu questions > > > I'm testing the total processing time of recon-all -all using -openmp > > 4,6,8,12 . > > The runs with 4 and 6 threads took almost the same amount of time. > > The runs with 8 and 12 threads took almost the same amount of time. > > > > (1) Is the work divided evenly between the threads (mri_ca_register, > > mri_en_register)? Or is 4 or 8 preferred and 6 and 12 don't buy you > > anything? > > > I can verify that the gain in processing time due to additional threads falls > off > rapidly with very little gain after 4. I dont know the exact times you got > from > your tests, and I havent done a time analysis in awhile, but we've never > noticed > a correlation preferring 4 and 8 threads over 6 and 12. However its an > interesting result and could very well be the case. > > > > (2) Is there a sync point in the software so that all the "other" > > threads wait for the slowest one to complete. > > > I dont do much of the openmp programming within freesurfer (only for > debugging purposes) but as far as I can tell all the openmp relevant code > blocks > occur at for-loop where threads are all created and diverge simultaneously and > then sync and at the same time at the conclusion of the for-loop. > > If you have an 8-12 processor machine and are looking to reduce your runtimes > you can take advantage of the fact that the left and right hemispheres can be > processed independently during the autorecon2 stage. > You could write a custom script along the lines of: > > recon-all -s subj -autorecon1 > recon-all -s subj -autorecon2-volonly > recon-all -s subj -autorecon2-perhemi -hemi rh -log recon-all-hemi rh.log - > notify rh-done.log -openmp 4 & > recon-all -s subj -autorecon2-perhemi -hemi lh -log recon-all-hemi-lh.log - > notify lh-done.log -openmp 4 & > while (rh-done.log AND lh-done.log do not both exist) > do > sleep 1 > end loop > recon-all -s -autorecon3 > > > > > Also, I tried a run with -use-gpu. > > I had to add a directory to LD_LIBRARY_PATH or I get an error on > > failure to find one of the libcuda's That is now ok but I get the > > following: > > > > Testing for CUDA device: > > ERROR: Unable to acquire a CUDA device! > > nvcc: NVIDIA (R) Cuda compiler driver > > Copyright (c) 2005-2012 NVIDIA Corporation Built on > > Fri_Sep_21_17:28:58_PDT_2012 Cuda compilation tools, release 5.0, > > V0.2.1221 > > > > Acquiring CUDA device > > Using default device > > Linux comet-30-08.sdsc.edu 3.10.73-1.el6.elrepo.x86_64 #1 SMP Thu Mar > > 26 > > 16:28:30 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux > > > > recon-all -s 173_1gpu exited with ERRORS at Thu Apr 30 09:17:42 PDT > > 2015_PATH but then I get the following error: > > > > Any thoughts would be welcome. > > > > Thanks, > > > > > Sorry but weve completely abandoned all support of CUDA in favor of openmp. > That said, people successfully have and still do use cuda with freesurfer > v5.3. > Assuming you have a cuda device compatible with the cuda libs shipped with > freesurfer it should simply work without you having to modify > LD_LIBRARY_PATH, but perhaps someone who has had success in this relm can > chime in. > > _______________________________________________ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > > The information in this e-mail is intended only for the person to whom it is > addressed. If you believe this e-mail was sent to you in error and the e-mail > contains patient information, please contact the Partners Compliance HelpLine > at http://www.partners.org/complianceline . If the e-mail was sent to you in > error but does not contain patient information, please contact the sender and > properly dispose of the e-mail. _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
