Yes, but the heartbeat coupling isn't necessary I think. One could even use ZK write/watch approach for faster assignment of regular work?
On Tue, Sep 24, 2013 at 2:24 PM, Steve Loughran <ste...@hortonworks.com> wrote: > On 21 September 2013 09:19, Sandy Ryza <sandy.r...@cloudera.com> wrote: > >> I don't believe there is any reason scheduling decisions need to be coupled >> with NodeManager heartbeats. It doesn't sidestep any race conditions >> because a NodeManager could die immediately after heartbeating. >> >> > historically its been done for scale: you don't need the JT reaching out to > 4K TT's just to give them work to do, instead let them connect in anyway > and get work that way. And once they start reporting in completion then > they can get given more work. It's very biased towards "worker nodes talk > to the master" over "master approaches workers" > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. -- Harsh J
