I am about to write a "cluster-lite" R solution for myself. I wanted to
know whether it already exists. If not, I will probably write up how I do
this, and I will make the code available.
Background: we have various linux and OSX systems, which are networked, but
not set up as a cluster. I have no one here to set up a cluster, so I need
a "hack" that facilitates parallel programming on standard networked
machines. I have accounts on all the machines, ssh access (of course
password-less), and networked file directory access.
what I am ultimately trying to accomplish is built around a "simple"
function, that my master program would invoke:
master.R:
multisystem( c("R slv.R 1 20 file1.out", "R slv.R 21 40 file2.out", "ssh
anotherhost R slv.R 41 80 file3.out"), announce=300)
multisystem() should submit all jobs simultaneously and continue only after
all are completed. it should also tell me every 300 seconds what jobs it is
still waiting for, and which have completed.
with basically no logic in the cluster, my master and slv programs have to
make up for it. master.R must have the smarts to know where it can spawn
jobs and how big each job should be. slv.R must have the smarts to place
its outputs into the marked files on the networked file directory. master.R
needs the smarts to combine the outputs of all jobs, and to resubmit jobs
that did not complete successfully. again, the main reason for doing all of
this is to avoid setting up a cluster across OSX and linux system, and still
to make parallel processing across linux/osx as easy as possible. I don't
think it gets much simpler than this.
now, I know how to write the multisystem() in perl, but not in R. so, if I
roll it myself, I will probably rely on a mixed R/perl system here. This is
not desirable, but it is the only way I know how to do this. if something
like multisystem() already exists in R native, please let me know and save
me from reinventing the wheel. if it does not, some perl/R combo for this
soon will.
regards,
/iaw
--
Ivo Welch (ivo.we...@brown.edu, ivo.we...@gmail.com)
[[alternative HTML version deleted]]
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
