the key thing would be to use different ZK paths for each cluster. You shouldn't need more than 2 ZK quorums even for a large (few thousand node) Hadoop clusters: one for the HA bits of the infrastructure (HDFS, YARN) and one for the applications to abuse. It's easy for apps using ZK to stick too much stuff, with too high a rate of change for ZK to be happy; overloading ZK can then impact those core infrastructure services. I don't believe that Spark is in the category of "antisocial" ZK Apps.
> On 22 Apr 2015, at 11:05, Sean Owen <so...@cloudera.com> wrote: > > Not that i've tried it, but, why couldn't you use one ZK server? I > don't see a reason. > > On Wed, Apr 22, 2015 at 7:40 AM, Akhil Das <ak...@sigmoidanalytics.com> wrote: >> It isn't mentioned anywhere in the doc, but you will probably need separate >> ZK for each of your HA cluster. >> >> Thanks >> Best Regards >> >> On Wed, Apr 22, 2015 at 12:02 AM, Michal Klos <michal.klo...@gmail.com> >> wrote: >>> >>> Hi, >>> >>> I'm trying to set up multiple spark clusters with high availability and I >>> was wondering if I can re-use a single ZK cluster to manage them? It's not >>> very clear in the docs and it seems like the answer may be that I need a >>> separate ZK cluster for each spark cluster? >>> >>> thanks, >>> M >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
