Andries, You are on-track with the first scenario. If you have the technical skills and time to install the Sun Grid Engine, or PBS (which does nearly the same thing), then the batch interface will be easy for others to submit as many 'recon-all' instances as they want and will handle balancing. Once a batch system is setup, you can expect to have to spend some time maintaining the system (help-line, monitoring for stalled jobs, failed nodes, or failed disks, etc). You could also write your own scripts implementing a simple batch system, if you don't mind semi-manually tracking the resources, and if you will oversee it or use it yourself, and keep it small-scale.
A fast connection between nodes is not necessary (at least for Freesurfer), since subjects are processed independently. The second scenario you mention is not really an option for large-scale subject studies, since even quad-processor machines could only realistically handle fours subjects at a time. Rather, the right/left hemi processing in parallel is something that could can be added transparently later, if, for instance, your nodes are multiprocessor machines but treated as a single node (as is the case with our cluster, where one processor on a dual-processor node is often unused). Tot ziens! Nick ps. no i am not dutch, but i worked in Amsterdam for a couple years at a start-up software company (SmartHaven BV) that went kaput. On Thu, 2006-12-07 at 12:48 +0100, Andries van der Leij wrote: > Nick and Uri: thank you for your replies, > > I'm not really into parallel processing and forgive me if I talk > nonsense :). From your stories I sketch the following possible > scenarios: > > 1. We use a sort of batch system. For instance, a Linux bash script with > 10 recon-all commands or with 10 FSL feat commands. The system assigns > each individual command to the node with the least load and the output > data are automatically stored on a central (fi RAID) server. I guess > this is the way the UCLA FSL/MAC grid works, since I can see no need to > alter the software itself, if the applications are suitable to work from > and write to an external computer. I'll contact the group exactly how > they work. I guess we need very fast connections between the individual > servers and the storage host to do this.... I don't know PBS, but I > think this is exactly how the Sun grid engine works also, so maybe they > are comparable. Batches are submitted to the host and are processed one > at a time. In fact this is how I work myself right now sometimes when > I'm in a hurry. I copy all my FSL source data to different computers and > let each computer run a subset of the feat batch. By hand I group the > data afterwards and I perform only the group averaging on a single > computer. > > 2. We allow for more sophisticated parallelizing. You give the example > of assigning the reconstruction of left and right hemispheres > separately. This is the most elegant solution but implies rewriting the > code. > > I think the second scenario would eventually be the most elegant but the > first scenario would be less difficult to implement. In fact, I guess > most applications (like FSL, freesurfer, SPM) work with batches, so if > we have software that manages these batches we already have won a lot > (and it saves a lot of manual work.... ;( > > What do you think? > > Thank you, > > Andries van der Leij > > PS: here's a presentation with screenshots of the sun grid: it looks > fairly straightforward. > > http://www.sun.com/products-n-solutions/edu/whitepapers/pdf/bioinformati > cs_supercomputer.pdf > > PS2: Nick, are you Dutch? > > > > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of U. > Hasson > Sent: Wednesday, December 06, 2006 9:50 PM > To: Nick Schmansky > Cc: Andries van der Leij; Freesurfer Mailing List > Subject: Re: [Freesurfer] Freesurfer and Grid computing > > At The Uni. of Chicago, we've been playing around with parallelizing > freesurfer on a 128 node grid (256 processors). Developers have > parallelized procedures (scripts) by unpacking "for" loops that > rotate across left and right hemispheres [i.e., they fork the > independent processing of left and right hemispheres to different > nodes running in parallel, whenever possible]. > > The main point of this work is to acquire provenance records and > therefore Freesurfer scripts are "wrapped" or expressed using a > virtual data system language (VDS/VDL). The freesurfer implementation, > AFAIK, is in its baby steps, but the general workflow model is pretty > well established > http://www.ci.uchicago.edu/wiki/bin/view/VDS/VDSWeb/WebMain > > Best, > Uri > > On 12/6/06, Uri Hasson <[EMAIL PROTECTED]> wrote: > > Here (Uni. of Chicago), we've been playing around with parallelizing > > freesurfer on a 128 node grid (256 processors), and developers have > > parallelized procedures (scripts) by unpacking "for" loops that > > rotate across left and right hemispheres [i.e., they fork the > > independent processing of left and right hemispheres to different > > nodes running in parallel, whenever possible]. > > > > The main point of this work is to acquire provenance records and > > therefore Freesurfer scripts are "wrapped" or expressed using a > > virtual data system language (VDS/VDL). The freesurfer implementation, > > AFAIK, is in its baby steps, but the general workflow model is pretty > > well established > > http://www.ci.uchicago.edu/wiki/bin/view/VDS/VDSWeb/WebMain > > > > Best, > > Uri > > > > > > On 12/6/06, Nick Schmansky <[EMAIL PROTECTED]> wrote: > > > Andries, > > > > > > I am not aware of usage of Freesurfer in a (Sun) Grid Engine > environment > > > (such as that used by the Cohen group at UCLA). > > > > > > However, here at the MGH/MIT/HMS Martinos Center we use a cluster of > > > some 100+ nodes configured with Linux Centos 4, and governed by PBS > > > (Portable Batch System). Researchers here often conduct studies > with > > > dozens to hundreds of brains, and for each subject, an instance of > > > Freesurfer's 'recon-all -s <subject> -all' script is submitted to > the > > > batch system, which, under the hood, gets submitted to one computing > > > node. Thus, several dozen brains can be processed in a day (and a > > > half). > > > > > > Freesurfer does not currently support fine-grain parallelism. Some > > > coarse-grain parallelism, whereby each brain hemisphere is processed > > > independently (benefiting multiprocessor nodes) is possible, but not > > > currently implemented in our 'recon-all' script, as the error > handling > > > and logging for doing so is somewhat tricky (and so this feature is > in- > > > the-works-but-not-anytime-soon). > > > > > > In short, if you plan on using Freesurfer in studies with large > numbers > > > of subjects, I would recommend some kind of computing cluster, and > some > > > fairly simple batch software (like PBS) should be sufficient. For > > > instance, I know of one group that has successfully run Freesurfer > on > > > their Altix Itanium Linux cluster. > > > > > > Groetjes, > > > > > > Nick > > > > > > > > > On Wed, 2006-12-06 at 18:44 +0100, Andries van der Leij wrote: > > > > > > > > > > > > > > > > > > > > > > > > > ______________________________________________________________________ > > > > > > > > From: Andries van der Leij > > > > Sent: Wednesday, December 06, 2006 5:59 PM > > > > To: 'freesurfer@nmr.mgh.harvard.edu' > > > > Subject: Freesurfer and Grid computing > > > > > > > > > > > > > > > > > > > > Dear Freesurfer community, > > > > > > > > > > > > > > > > I'm a PHD student at the university of Amsterdam and I'm currently > > > > investigating the possibilities to streamline our MRI data > processing > > > > stream. Next summer we'll obtain a research-only scanner. I'm > trying > > > > to push the group to also invest in computing power and am > currently > > > > investigating the applications that researchers will most probably > > > > use. > > > > > > > > > > > > > > > > I came across a project of the group of Cohen at UCLA. They have > > > > configured a Apple (unix) grid and have proposed a more or less > > > > standard setup specially designed for MRI analyses: > > > > > > > > > > > > > > > > http://airto.bmap.ucla.edu/mt- > > > > static/NICluster/archives/2005/06/welcome.html > > > > > > > > > > > > > > > > It is my understanding that one of the members has rewritten the > FSL > > > > code which allow distributed parallel processing in a Grid. See > the > > > > benchmarks here: > > > > > > > > > > > > > > > > > http://airto.bmap.ucla.edu/bmcweb/bmc_bios/MarkCohen/Apple/Benchmarks.ht > m > > > > > > > > > > > > > > > > My question is fairly simple: Are similar steps taken in the > > > > Freesurfer community? I have no experience with this app myself, > but > > > > it is my understanding that Freesurfer consumes a lot of > resources. > > > > > > > > > > > > > > > > Thank you very much in advance, > > > > > > > > > > > > > > > > Andries van der Leij > > > > > > > > > > > > _______________________________________________ > > > > Freesurfer mailing list > > > > Freesurfer@nmr.mgh.harvard.edu > > > > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > > > > > _______________________________________________ > > > Freesurfer mailing list > > > Freesurfer@nmr.mgh.harvard.edu > > > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > > > > > > _______________________________________________ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > _______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
