Re: Apache Spark 3.2 Expectation

Hi all, Just as a gentle reminder, I will do the branch cut tomorrow. Please focus on finalizing the works to land in Spark 3.2.0. After the branch cut, we can still merge the ongoing major features mentioned in this thread. There should no be other new features in branch 3.2. Thanks! On Thu, Jun

Re: Spark on Kubernetes scheduler variety

Hi Klaus, Thanks https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/1289 view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or

what's the potential security issue of reusing a spark session for multi users

Spark Devs, what's the potential security issue to reuse one spark session for running spark-hive queries, spark jobs ? suppose the session usage is isolated to different time chunks , and some clean up is done before each usage (for example hive cache is recreated before each usage). thanks.

Re: Spark on Kubernetes scheduler variety

Hi Mich, Would you help to open an issue at spark-on-k8s-operator repo? We're going to submit a PR to update the install steps :) -- Klaus On Wed, Jun 30, 2021 at 12:24 AM Mich Talebzadeh wrote: > Hi Yikun > > In reference > > > https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob

Re: Spark on Kubernetes scheduler variety

Hi Michel, Thanks for the link. I am familiar with G-Research as I met them in my presentation in London back in October 2019. The amanda project sems to create super-scheduling on top of Kubernetes clusters and I quote: "Armada is an application to achieve high throughput of run-to-completion