you are right. I was totally confused at the beggining. Thanks a lot. 2009/3/4, Vincent Schut <sc...@sarvision.nl>: > hugo rivera wrote: > > > The cluster has torque installed as the resource manager. I think it > > runs of top of pbs (an older project). > > As far as I know now I just have to call a qsub command to submit my > > jobs on a queue, then the resource manager allocates a processor in > > the cluster for my process to run till is finished. > > > > Well, I don't know torque neither pbs, but I'm guessing that when you > submit a job, this job will be some program or script that is run on the > allocated processor? If so, your initial question of forking vs threading is > bogus. Your cluster manager will run (exec) your job, which if it is a > python script will start a python interpreter for each job. I guess that's > the overhead you get when running a flexible cluster system, flexible > meaning that it can run any type of job (shell script, binary executable, > python script, perl, etc.). > However, your overhead of starting new python processes each time may seem > significant when viewed in absolute terms, but if each job processes lots of > data and takes, as you said, 5 min to run on a decent processor, don't you > think the startup time for the python process would become non-significant? > For example, on a decent machine here, the first time python takes 0.224 > secs to start and shutdown immediately, and consequetive starts take only > about 0.009 secs because everything is still in memory. Let's take the 0.224 > secs for a worst case scenario. That would be approx 0.075 percent of your > job execution time. Now lets say you have 6 machines with 8 cores each and > perfect scaling, all your jobs would take 6000 / (6*8) *5min = 625 minutes > (10 hours 25 mins) without python starting each time, and 625 minutes and 28 > seconds with python starting anew each job. Don't you think you could just > live with these 28 seconds more? Just reading this message might already > have taken you more than those 28 seconds... > > Vincent. > > > > > And I am not really sure if I have access to all the nodes, so I can > > install pp on each one of them. > > > > 2009/3/4, Vincent Schut <sc...@sarvision.nl>: > > > > > hugo rivera wrote: > > > > > > > > > > Thanks for the advice. > > > > Nevertheless I am in no position to decide what pieces of software the > > > > cluster will run, I just have to deal with what I have, but anyway I > > > > can suggest other possibilities. > > > > > > > > > > > Well, depends on how you define 'software the cluster will run'. Do you > > > mean cluster management software, or really any program or script or > python > > > module that needs to be installed on each node? Because for pp, you > won't > > > need any cluster software. pp is just some python module and helper > scripts. > > > You *do* need to install this (pure python) module on each node, yes, > but > > > that's it, nothing else needed. > > > Btw, you said 'it's a small cluster, about 6 machines'. Now I'm not an > > > expert, but I don't think you can do threading/forking from one machine > to > > > another (on linux). So I suppose there already is some cluster > management > > > software involved? And while you appear to be "in no position to decide > what > > > pieces of software the cluster will run", you might want to enlighten us > on > > > what this cluster /will/ run? Your best solution might depend on that... > > > > > > Cheers, > > > Vincent. > > > > > > > > > > > > > > > > > > > > >
-- Hugo
