Can't we use zoo keeper for leader election in Cassandra and based on who is leader ..run azkaban or any app instance for that matter on that Cassandra server. I'm thinking that I can copy the applocation folder to all nodes and then determine which one to run using zookeeper. Is that possible ?
Sent from Outlook On Sun, Aug 16, 2015 at 6:47 AM -0700, "John Wong" <gokoproj...@gmail.com> wrote: Hi I am not familiar with Azkaban and probably a better question to the Azkaban community IMO. But there seems to be two modes (http://azkaban.github.io/azkaban/docs/2.5/) one is solo and one is two-server mode, but either way I think still SPOF? If there is no election, just based on process, my 2 cents would be monitor, alert, and start the process somewhere else. Better yet, don't install the process on Cassandra node. Keep your instance for one purpose only. If you run cloud like AWS you will be able to autoscale min1 max1 easily. Note: In peer-to-peer architecture, there is simply no concept of master. You can start with some seed nodes for discovery. It depends how you design discovery. On Sat, Aug 15, 2015 at 11:49 AM, Vikram Kone <vikramk...@gmail.com> wrote: Hi, We are planning to install Azkaban in solo server mode on a 24 node cassandra cluster to be able to schedule spark jobs with intricate dependency chain. The problem, is since Cassandra has a no-SPOF architecture ie any node can become the master for the cluster, it creates the problem for Azkaban master since it's not a peer-peer architecture where any node can become the master. Only a single mode has to be master at any given time. What are our options here? Are there any framworks or tools out there that would allow any application to run on a cluster of machines with high availablity? Should I be looking at something like zookeeper for this ? Or Mesos may be?
