Re: Spark on Kubernetes scheduler variety

2021-06-23 Thread Klaus Ma
Hi team, I'm kube-batch/Volcano founder, and I'm excited to hear that the spark community also has such requirements :) Volcano provides several features for batch workload, e.g. fair-share, queue, reservation, preemption/reclaim and so on. It has been used in several product environments with Sp

Re: Spark on Kubernetes scheduler variety

2021-06-23 Thread Mich Talebzadeh
Please allow me to be diverse and express a different point of view on this roadmap. I believe from a technical point of view spending time and effort plus talent on batch scheduling on Kubernetes could be rewarding. However, if I may say I doubt whether such an approach and the so-called democra