On Apr 28, 2010, at 5:04 AM, Steve Loughran wrote: > prajyot bankade wrote: >> Hello Everyone, >> I have just started reading about hadoop job tracker. In one book I read >> that there is only one job tracker who is responsible to distribute task to >> worker system. Please make me right if i say some thing wrong. >> I have few questions, >> why there is only one job tracker? > > to provide a single place to make scheduling decisions > (thread hijack)
Why is this an advantage? (I mean, I know it's an advantage in terms of the current architecture... just indulging in some blue-sky thinking here). One of the projects I work with is the Condor Project out of Madison: http://www.cs.wisc.edu/condor/ who have been building a distributed computing infrastructure for about 20 years. Here is one of my favorite "overview" papers of theirs: http://www.cs.wisc.edu/condor/doc/condor-practice.pdf (my favorite is sections 4, 5, and 6.2) They have gotten lots of mileage out of breaking the scheduling and the resource provision into two different components. Having multiple jobtrackers would be very advantageous if it didn't require you to partition your pool. One use case would be to separate out the "production work" from "research activities". You could have a 'production jobtracker' which is accessible to a small number of users and has "known good", pre-approved, business-critical workflows and a 'research jobtracker' which more folks are allowed to use without pre-approved workflows. This way, if a researcher accidentally crashes the jobtracker in the middle of the night, your business-critical work continues. There's plenty of merit to the idea and worth thinking about. Brian
smime.p7s
Description: S/MIME cryptographic signature
