Re: Spark on Kubernetes scheduler variety

Hi Mich, Would you help to open an issue at spark-on-k8s-operator repo? We're going to submit a PR to update the install steps :) -- Klaus On Wed, Jun 30, 2021 at 12:24 AM Mich Talebzadeh wrote: > Hi Yikun > > In reference > > > https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob

Re: Spark on Kubernetes scheduler variety

Hi team, I'm kube-batch/Volcano founder, and I'm excited to hear that the spark community also has such requirements :) Volcano provides several features for batch workload, e.g. fair-share, queue, reservation, preemption/reclaim and so on. It has been used in several product environments with Sp

Re: Build customized resource manager

wrote: > I don't know if it all works but some work was done to make cluster > manager pluggable, see SPARK-13904. > > Tom > > On Wednesday, November 6, 2019, 07:22:59 PM CST, Klaus Ma < > klaus1982...@gmail.com> wrote: > > > Any suggestions? > > - Klau

Re: Build customized resource manager

Any suggestions? - Klaus On Mon, Nov 4, 2019 at 5:04 PM Klaus Ma wrote: > Hi team, > > AFAIK, we built k8s/yarn/mesos as resource manager; but I'd like to did > some enhancement to them, e.g. integrate with Volcano > <http://github.com/volcano-sh/volcano> in k8s. Is

Build customized resource manager

Hi team, AFAIK, we built k8s/yarn/mesos as resource manager; but I'd like to did some enhancement to them, e.g. integrate with Volcano in k8s. Is that possible to do that without fork the whole spark project? For example, enable customized resource manager wi