Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

@Weiwei @Chenya > Thanks for bringing this up. This is quite interesting, we definitely should participate more in the discussions. Thanks for your reply and welcome to join the discussion, I think the input from Yunikorn is very critical. > The main thing here is, the Spark community should mak

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

IIUC, PodGroup is only supported in Volcano, this is not a common API adopted by K8s, at least not today. Spark needs to be agnostic about the schedulers, as an example, when we run Spark on YARN, does Spark need to know if that is FairScheduler or CapacityScheduler? IMO, we should build things gen

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Hi, Well, in mitigation, one cannot address all the available scheduler options at once. Certainly PodGroup is an option, unless there are reasons to believe that this is not a right choice. Therefore, I stand corrected, I fail to see where problematic comes into it, unless you may care to elabora

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Hi Chenya Thanks for bringing this up. This is quite interesting, we definitely should participate more in the discussions. The main thing here is, the Spark community should make Spark pluggable in order to support other schedulers, not just for Volcano. It looks like this proposal is pushing rea

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Great to have other integrations and improved K8s support.  Left some comments/questions in the design doc.  TomOn Tuesday, November 30, 2021, 02:46:42 AM CST, Yikun Jiang wrote: Hey everyone, I'd like to start a discussion on "Support Volcano/Alternative Schedulers Proposal". This

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

This is a great feature/idea. I'd love to get involved in some form (testing and/or documentation). This could be my 1st contribution to Spark! On Tue, Nov 30, 2021 at 10:46 PM John Zhuge wrote: > +1 Kudos to Yikun and the community for starting the discussion! > > On Tue, Nov 30, 2021 at 8:47 A

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

+1 Kudos to Yikun and the community for starting the discussion! On Tue, Nov 30, 2021 at 8:47 AM Chenya Zhang wrote: > Thanks folks for bringing up the topic of natively integrating Volcano and > other alternative schedulers into Spark! > > +Weiwei, Wilfred, Chaoran. We would love to contribute

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Thanks folks for bringing up the topic of natively integrating Volcano and other alternative schedulers into Spark! +Weiwei, Wilfred, Chaoran. We would love to contribute to the discussion as well. >From our side, we have been using and improving on one alternative resource scheduler, Apache Yuni

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Thanks for putting this together, I’m really excited for us to add better batch scheduling integrations. On Tue, Nov 30, 2021 at 12:46 AM Yikun Jiang wrote: > Hey everyone, > > I'd like to start a discussion on "Support Volcano/Alternative Schedulers > Proposal". > > This SPIP is proposed to mak

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Hi, Good points. I have started making some suggestions to the Design doc as referenced above. My main concern with the current scheduling is the fact that there seems to be one pod per node association with k8s application. The model works on the basis of the "one-container-per-Pod" model

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Adding @Holden Karau @Dongjoon Hyun @wuyi FYI On Tue, 30 Nov 2021 at 17:46, Yikun Jiang wrote: > Hey everyone, > > I'd like to start a discussion on "Support Volcano/Alternative Schedulers > Proposal". > > This SPIP is proposed to make spark k8s schedulers provide more YARN like > features (s

[DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Hey everyone, I'd like to start a discussion on "Support Volcano/Alternative Schedulers Proposal". This SPIP is proposed to make spark k8s schedulers provide more YARN like features (such as queues and minimum resources before scheduling jobs) that many folks want on Kubernetes. The goal of this