Hi I am not familiar with Azkaban and probably a better question to the Azkaban community IMO. But there seems to be two modes ( http://azkaban.github.io/azkaban/docs/2.5/) one is solo and one is two-server mode, but either way I think still SPOF? If there is no election, just based on process, my 2 cents would be monitor, alert, and start the process somewhere else. Better yet, don't install the process on Cassandra node. Keep your instance for one purpose only. If you run cloud like AWS you will be able to autoscale min1 max1 easily.
Note: In peer-to-peer architecture, there is simply no concept of master. You can start with some seed nodes for discovery. It depends how you design discovery. On Sat, Aug 15, 2015 at 11:49 AM, Vikram Kone <vikramk...@gmail.com> wrote: > Hi, > We are planning to install Azkaban in solo server mode on a 24 > node cassandra cluster to be able to schedule spark jobs with intricate > dependency chain. The problem, is since Cassandra has a no-SPOF > architecture ie any node can become the master for the cluster, it creates > the problem for Azkaban master since it's not a peer-peer architecture > where any node can become the master. Only a single mode has to be master > at any given time. > > What are our options here? Are there any framworks or tools out there that > would allow any application to run on a cluster of machines with high > availablity? > Should I be looking at something like zookeeper for this ? Or Mesos may > be?
