Hi, Good points. I have started making some suggestions to the Design doc as referenced above.
My main concern with the current scheduling is the fact that there seems to be one pod per node association with k8s application. The model works on the basis of the "one-container-per-Pod" model <https://kubernetes.io/docs/concepts/workloads/pods/> meaning that for each node of the cluster you will have one node running the driver and each remaining node running one executor each. I tested this using Google Kubernetes cluster (GKE) with dynamic adjustment of the number of nodes. Now there is also the lead time for the driver to create the executors. when the number of executors set to a higher number of the nodes through --conf spark.executor.instances=$NEXEC, For example I am testing this in Google Kubernetes Cluster with three nodes and no of executors set at 9 NAME READY STATUS RESTARTS AGE randomdatabigquery-fbfaa97d7019ffa7-exec-1 0/1 ContainerCreating 0 22s randomdatabigquery-fbfaa97d7019ffa7-exec-2 0/1 ContainerCreating 0 22s randomdatabigquery-fbfaa97d7019ffa7-exec-3 1/1 Running 0 22s randomdatabigquery-fbfaa97d7019ffa7-exec-4 0/1 ContainerCreating 0 22s randomdatabigquery-fbfaa97d7019ffa7-exec-5 0/1 ContainerCreating 0 21s randomdatabigquery-fbfaa97d7019ffa7-exec-6 1/1 Running 0 20s randomdatabigquery-fbfaa97d7019ffa7-exec-7 0/1 ContainerCreating 0 20s randomdatabigquery-fbfaa97d7019ffa7-exec-8 0/1 ContainerCreating 0 20s randomdatabigquery-fbfaa97d7019ffa7-exec-9 0/1 Pending 0 20s sparkbq-ad80717d701955ad-driver 1/1 Running 0 66s Some of these exectors are totally redundant but the driver still takes time to create them which I believe is a waste of resources. This needs to be avoided. Secondally, the current scheduler is based on Spark standalone mode, which I think is less strict than the Yarn mode. I am not sure how much volcano contribution is to make this better. However, we need to improve the current scheduling regardless. HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Tue, 30 Nov 2021 at 08:47, Yikun Jiang <yikunk...@gmail.com> wrote: > Hey everyone, > > I'd like to start a discussion on "Support Volcano/Alternative Schedulers > Proposal". > > This SPIP is proposed to make spark k8s schedulers provide more YARN like > features (such as queues and minimum resources before scheduling jobs) that > many folks want on Kubernetes. > > The goal of this SPIP is to improve current spark k8s scheduler > implementations, add the ability of batch scheduling and support volcano as > one of implementations. > > Design doc: > https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg > JIRA: https://issues.apache.org/jira/browse/SPARK-36057 > Part of PRs: > Ability to create resources https://github.com/apache/spark/pull/34599 > Add PodGroupFeatureStep: https://github.com/apache/spark/pull/34456 > > Regards, > Yikun >
