I've done quite a bit of this nested parallelism with BatchJobs/SGE over multicore.
I daresay I haven't looked at BiocParallel as the solutions seemed pretty obvious with standard calls. Should BiocParallel have a vignette that indicates added value? I would like us to have a worked demonstration in an AMI for a nested parallel solution. The "Cloud-enabled cis-eQTL..." workflow document gives some gory details for snow/multicore in EC2 using starcluster to manage the AMIs, and identifies an AMI where the process can be demonstrated. I would like to control the process via BatchJobs but I haven't been able to get SGE running in EC2 ... On Fri, Feb 14, 2014 at 12:28 PM, Michael Lawrence < lawrence.mich...@gene.com> wrote: > On Fri, Feb 14, 2014 at 9:20 AM, Laurent Gatto <lg...@cam.ac.uk> wrote: > > > > > Dear all, > > > > Looking back in my (sparse) notes from the last European Bioc meeting, I > > saw two points that were mentioned during the BatchJobs and BiocParallel > > talk that are of particular interest to me. > > > > - BatchJobs: is there support, or are there any plans to support the > > HTCondor infrastructure [1], and will this back-end be supported in > > BiocParallel? > > > > > If BatchJobs supports it, then so will BiocParallel, thanks to the > abstraction. > > > > - BiocParallel: is there support, or are there any plans to support > > nested parallelisation. The use case I have in mind is to have a first > > level of parallelisation to distribute n sets jobs on n different > > machines using snow for example, and then each set of jobs would be > > distributed on their respective machine via multicore? > > > > > Yes, this is done by passing a list of BiocParallelParam objects to > bplapply. Not sure if it's ever been tested though. > > > > Any comments on the above solutions or their feasibility would be very > > welcome. > > > > Thank you very much in advance. > > > > Best wishes, > > > > Laurent > > > > [1] http://research.cs.wisc.edu/htcondor/ > > > > > > -- > > Laurent Gatto > > http://cpu.sysbiol.cam.ac.uk > > http://proteomics.bio.cam.ac.uk > > > > _______________________________________________ > > Bioc-devel@r-project.org mailing list > > https://stat.ethz.ch/mailman/listinfo/bioc-devel > > > > [[alternative HTML version deleted]] > > _______________________________________________ > Bioc-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/bioc-devel > [[alternative HTML version deleted]] _______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
