On 08/13/2016 08:24 PM, guyoh wrote: > My company is trying to decide whether to use kubernetes or mesos. Since we > are planning to use Spark in the near future, I was wandering what is the > best choice for us. > Thanks, > Guy > Both Kubernetes and Mesos enables you to share your infrastructure with other workloads. In K8s it is mainly standalone mode and to provide failure recovery you can use the options mentioned here https://spark.apache.org/docs/latest/spark-standalone.html#high-availability I have tested file system recovery with K8s and it works fine, as K8s restart the master with in few seconds. In K8s you can dynamically scale your cluster as K8s support horizontal pod autoscaling http://blog.kubernetes.io/2016/03/using-Spark-and-Zeppelin-to-process-Big-Data-on-Kubernetes.html which helps you spin down your spark cluster when not is use and you can run other workloads using docker/rkt containers on your resources and scale up spark cluster when needed given you have free resources. You don't need static partitioning for your spark cluster when running on k8s, as spark will run in containers and will share the same underlying resources as others.
You can soonish (https://github.com/apache/spark/pull/13950) use the authentication based on Github,Google,FB etc to access Spark master UI when deployed in standalone e.g. on K8s and only expose the master UI to access all other UIs (worker logs, app). As usual choice is yours, I just wanted to give you the info about k8s side. - Gurvinder > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/mesos-or-kubernetes-tp27530.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
