Hi Timothée, I had the same dilema and I'm betting on spawning workers on Docker containers through SSH, similar to your 2nd suggestion but with Docker support. AFAIK, the main drawback of not using batch schedulers is the lack of: fault tolerance mechanisms, queue management, and support for accurate/detailed job monitoring.
The main reason for using Docker/SSH is that I wanted to use as few framework (software layers) as possible for prototyping. Actually, that's why I'm using Julia: I can quickly prototype distributed systems with few (or even any) frameworks, batch systems, and so forth. Will you use DistributedArrays at these machines? If not, which kind of distributed support do you plan to use? André Lage. On Monday, October 12, 2015 at 4:36:33 PM UTC-3, Timothée Poisot wrote: > > Hi List, > > I am starting to deploy a small cluster of machines, and I'd like to > take advantage of Julia's ability for distributed computing. I have one > "big" machine purely for computing (48 Xeon cores, 256 GB RAM), and a > few workstations (12 i7 cores, 64 GB RAM). This is all connected on a > "head" machine, which also routes our local traffic onto the university > network. > > All of this runs Fedora 22 (server or workstation), all machines have > the same versions of the same libraries, and everyone has accounts on > all machines. > > On a typical day, students use their workstations, but there are most > likely a few available cores. I know I can add workers over ssh, but > I'm wondering what the best strategy is. > > 1. Should I use a job manager on our head machine? > 2. Should I spawn workers on the other workstations directly? > 3. Some other solution? > > In short, I'd be really interested in discussing how you handled > similar situations. > > t > > -- > Timothée Poisot, PhD > > Professeur adjoint > Département des sciences biologiques > Université de Montréal > > phone : 514 343-7691 > web : http://poisotlab.io > twitter: @PoisotLab > meeting: https://tpoisot.youcanbook.me/ > >
