It is as Jagat said. The Masters do not need to know about one another, as ZooKeeper manages their implicit communication. As for Workers (and applications, such as spark-shell), once a Worker is registered with *some *Master, its metadata is stored in ZooKeeper such that if another Master is elected, it informs all current Workers (and applications), so they need not have a priori knowledge of all future Masters.
On Wed, Apr 9, 2014 at 3:46 PM, Dmitriy Lyubimov <dlie...@gmail.com> wrote: > ah. > > standalone HA master was added in 0.9.0. Same logic, but Spark-native. > > > On Wed, Apr 9, 2014 at 3:31 PM, Pradeep Ch <pradeep.chanum...@gmail.com>wrote: > >> Thanks Dmitriy. But I want multi master support when running spark >> standalone. Also I want to know if this multi master thing works if I use >> spark-shell. >> >> >> On Wed, Apr 9, 2014 at 3:26 PM, Dmitriy Lyubimov <dlie...@gmail.com>wrote: >> >>> The only way i know to do this is to use mesos with zookeepers. you >>> specify zookeeper url as spark url that contains multiple zookeeper hosts. >>> Multiple mesos masters are then elected thru zookeeper leader election >>> until current leader dies; at which point mesos will elect another master >>> (if still left). >>> >>> iirc, in this mode spark master never runs, only master slaves are being >>> spun by mesos slaves directly. >>> >>> >>> >>> >>> >>> On Wed, Apr 9, 2014 at 3:08 PM, Pradeep Ch >>> <pradeep.chanum...@gmail.com>wrote: >>> >>>> Hi, >>>> >>>> I want to enable Spark Master HA in spark. Documentation specifies that >>>> we can do this with the help of Zookeepers. But what I am worried is how to >>>> configure one master with the other and similarly how do workers know that >>>> the have two masters? where do you specify the multi-master information? >>>> >>>> Thanks for the help. >>>> >>>> Thanks, >>>> Pradeep >>>> >>> >>> >> >
