Hi, Is it possible to start a spark standalone master inside my own JVM?
What I would like to do is the following:- in my own main (object MyApp extends App):- * Start zookeeper in embedded (and clustered) mode * Start a spark master in same jvm referring to the above zookeeper quorum for the HA (standby) behaviour * Start spark worker in same jvm managed by the spark master quorum started above * Create a spark context against the above embedded master quorum in the same jvm I would then want to deploy many such "all in one" apps across a cluster injecting appropriate (hopefully symmetric) configuration. I understand that adding a new "node" like the the above to an existing cluster would probably require some global reconfiguration (mostly because of a new Zk and Master being added). *An alternative* might be that I start spark masters and zk instances externally, but I'd still want the the workers and the SparkContext to be in a single jvm, that way I (can I?) can dynamically add workers (and SparkContexts) to the same "spark cluster" without master reconfiguration. job submission would happen via any of the open SparkContexts, possibly coordinated through something like hazelcast/zookeeper Pointers to sample code would be very helpful. A debate around Pros/Cons of taking this approach would be really useful also. Regards, -- Aditya Varun Chadha | http://www.adichad.com
