Hi Darren,Nick and freesurfers, Thanks very much for your replies! I've discussed this and have a couple of questions. I would be very glad if you could help me out.
I'm currently talking with my group to decide where to go. Like I said, we will get a research-only 3T scanner, which (if used as much as we hope) will deliver about 20 GB of raw data each week. We need to think of a solid processing and storage pipeline in order to be able to do something with the data ;) We discussed this and are trying to come up with a good solution, both storage and analyses-wise. One solution for the enhancement of the analyses would be to buy a couple of solid workstations. The other would be a grid solution. The main problem isn't as much finance as it is manpower. No current member of the faculty has the time to invest in setting up and maintaining a grid if that will be a very difficult task to do. Hopefully we will get the money to hire an ICT person to do this. However, this is not certain. So I have a question concerning the difficulty of setting up the grid like proposed by Nick: semi-parallel in the sense that different commands of a batch are distributed to different nodes. We are wondering just how difficult this is. I address this question to both of you. Darren: you have SUN grid and Nick you have something like SUN grid. Nick, you state: <snip> You are on-track with the first scenario. If you have the technical skills and time to install the Sun Grid Engine, or PBS (which does nearly the same thing), then the batch interface will be easy for others to submit as many 'recon-all' instances as they want and will handle balancing. Once a batch system is setup, you can expect to have to spend some time maintaining the system (help-line, monitoring for stalled jobs, failed nodes, or failed disks, etc). You could also write your own scripts implementing a simple batch system, if you don't mind semi-manually tracking the resources, and if you will oversee it or use it yourself, and keep it small-scale. </snip> The issue of course is this: 1. Just how much technical skills and time does the installation and maintenance of a Grid cost? It is my impression that the implementation of the SUN grid is not that hard. But I'm always an optimist if it comes to computers, although at times this has proven to be a great overestimation of my own knowledge and skills. 2. How 'intuitive' is the usage of a batch system like SUN grid and how stable is it? That is: given that we write guidelines, howtos, and set up a Wiki, how difficult will drag&drop type users will be able to work with such a system? The final maintenance issues that Nick addresses concerns failure. 3. Hardware failure: freezing nodes and disk failures I expect to cost as much time as it will cost when we buy individual workstations, Maybe even less? 4. Software failure. Maybe I'm way off here, but it's my impression that the host running the batch system uploads the input data from a central server to a node, which analyses the data and then downloads the resulting output again, deleting the data on the node. If this is correct and if a single job fails, do you have to manually delete working data on a node? I hope you can help me out with this. Thank you! Andries -----Original Message----- From: Darren Weber [mailto:[EMAIL PROTECTED] Sent: Friday, December 08, 2006 3:33 AM To: Andries van der Leij Cc: freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] Freesurfer and Grid computing -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 We have a SUN grid system at UCSF. I have some shell scripts for using freesurfer on this type of grid. As said earlier, freesurfer does not contain multi-threaded programs. Use of the grid is simply to run multiple instances of freesurfer programs on different processing nodes (computers). The grid is managed by scheduling programs. We submit jobs to those scheduling programs (qsub in our case). My simple solution consists of two parts, a) a subject specific script to run recon-all (only if necessary) and b) another script to find and loop over subjects and submit those jobs to the grid. a) freesurfer_recon_script.csh (Note that all of my freesurfer environment variables are setup in ~/.cshrc and this environment is configured to work correctly on all nodes of the grid system, providing the NFS disk mapping is working.) ############### CUT BELOW HERE ################# #!/bin/csh echo $0 $1 cd $SUBJECTS_DIR pwd if ( "$1" == "" ) then echo echo "Missing <subjID>, please use" echo "$0 <subjID>" echo exit 1 endif if ( -d $1 ) then echo $1 # has this been done already? if ( -s $1/surf/lh.pial && -s $1/surf/rh.pial ) then echo echo "The pial surfaces already exist:" echo ls -l $1/surf/* echo else recon-all -subjid $1 -all endif endif ############### CUT ABOVE HERE ################# b) ~/bin/freesurfer_recon_gridSubmit.csh ############### CUT BELOW HERE ################# #!/bin/csh cd $SUBJECTS_DIR foreach sub ( `ls -d ucsf_??` ) echo $sub # has this been done already? if ( -s $1/surf/lh.pial && -s $1/surf/rh.pial ) then echo echo "The pial surfaces already exist." echo else qsub -q dnl.q -l arch=lx24-x86 ~/bin/freesurfer_recon_script.csh $sub & endif end ############### CUT ABOVE HERE ################# It's a little bit clumsy this way, but qsub is not really intelligent about handling job specifications (or I am too stupid to figure out how to get it all into one script!), so I ask it to run a shell script. Best, Darren Andries van der Leij wrote: > > > > > ------------------------------------------------------------------------ > > *From:* Andries van der Leij > *Sent:* Wednesday, December 06, 2006 5:59 PM > *To:* 'freesurfer@nmr.mgh.harvard.edu' > *Subject:* Freesurfer and Grid computing > > > > Dear Freesurfer community, > > > > I?m a PHD student at the university of Amsterdam and I?m currently > investigating the possibilities to streamline our MRI data processing > stream. Next summer we?ll obtain a research-only scanner. I?m trying to > push the group to also invest in computing power and am currently > investigating the applications that researchers will most probably use. > > > > I came across a project of the group of Cohen at UCLA. They have > configured a Apple (unix) grid and have proposed a more or less standard > setup specially designed for MRI analyses: > > > > http://airto.bmap.ucla.edu/mt-static/NICluster/archives/2005/06/welcome. html > > > > It is my understanding that one of the members has rewritten the FSL > code which allow distributed parallel processing in a Grid. See the > benchmarks here: > > > > http://airto.bmap.ucla.edu/bmcweb/bmc_bios/MarkCohen/Apple/Benchmarks.ht m > > > > My question is fairly simple: Are similar steps taken in the Freesurfer > community? I have no experience with this app myself, but it is my > understanding that Freesurfer consumes a lot of resources. > > > > Thank you very much in advance, > > > > Andries van der Leij > > > ------------------------------------------------------------------------ > > _______________________________________________ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer - -- Darren L. Weber, Ph.D. Postdoctoral Scholar Dynamic Neuroimaging Laboratory, UCSF Department of Radiology, 185 Berry Street, Suite 350, Box 0946, San Francisco, CA 94107, USA. Tel: +1 415 353-9444 Fax: +1 415 353-9421 www: http://dnl.ucsf.edu/users/dweber "To explicate the uses of the brain seems as difficult a task as to paint the soul, of which it is commonly said, that it understands all things but itself." Thomas Willis (The Anatomy of the Brain and Nerves, 1664) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFeM7yxaCYN7qs0v4RAvIuAJ0dUPOOWNuHpkhgZT3XZPMGjcgHagCdFoIM eJ8OPogaXktVZyW0CLajGXA= =ts8p -----END PGP SIGNATURE----- _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
