Re: [UPDATE] Apache Spark 3.1.0 Release Window

I think you put Jan 2020 instead of 2021 :-) Le mar. 13 oct. 2020 à 00:51, Xiao Li a écrit : > Thank you, Dongjoon > > Xiao > > On Mon, Oct 12, 2020 at 4:19 PM Dongjoon Hyun > wrote: > >> Hi, All. >> >> Apache Spark 3.1.0 Release Window is adjusted like the following today. >> Please check the

Re: Spark3 on k8S reading encrypted data from HDFS with KMS in HA

oop KMS, and it worked well. > Somehow, I could not recall, if I had the kubernetes in the mix. Somehow, > seeing the error, it is not clear what caused the failure. Can I reproduce > this somehow? > > Thanks, > > On Sat, Aug 15, 2020 at 7:18 PM Michel Sumbul > wrote: &

Re: Spark3 on k8S reading encrypted data from HDFS with KMS in HA

d with KMS :-) Thanks, Michel Le jeu. 13 août 2020 à 14:32, Michel Sumbul a écrit : > Hi guys, > > Does anyone try Spark3 on k8s reading data from HDFS encrypted with KMS in > HA mode (with kerberos)? > > I have a wordcount job running with Spark3 reading data on HDFS (hadoop &

Spark3 on k8S reading encrypted data from HDFS with KMS in HA

Hi guys, Does anyone try Spark3 on k8s reading data from HDFS encrypted with KMS in HA mode (with kerberos)? I have a wordcount job running with Spark3 reading data on HDFS (hadoop 3.1) everything secure with kerberos. Everything works fine if the data folder is not encrypted (spark on k8s). If t

Re: Spark 3 pod template for the driver

Hello, Adding the dev mailing list maybe there is someone here that can help to have/show a valid/accepted pod template for spark 3? Thanks in advance, Michel Le ven. 26 juin 2020 à 14:03, Michel Sumbul a écrit : > Hi Jorge, > If I set that in the spark submit command it works but I w

Re: Spark 3 pod template for the driver

Hi Jorge, If I set that in the spark submit command it works but I want it only in the pod template file. Best regards, Michel Le ven. 26 juin 2020 à 14:01, Jorge Machado a écrit : > Try to set spark.kubernetes.container.image > > On 26. Jun 2020, at 14:58, Michel Sumbul wrote: &g

Spark 3 pod template for the driver

Hi guys, I try to use Spark 3 on top of Kubernetes and to specify a pod template for the driver. Here is my pod manifest or the driver and when I do a spark-submit with the option: --conf spark.kubernetes.driver.podTemplateFile=/data/k8s/podtemplate_driver3.yaml I got the error message that I n

Spark 3 pod template for the driver

Hi guys, I try to use Spark 3 on top of Kubernetes and to specify a pod template for the driver. Here is my pod manifest or the driver and when I do a spark-submit with the option: --conf spark.kubernetes.driver.podTemplateFile=/data/k8s/podtemplate_driver3.yaml I got the error message that I ne

Re: Exact meaning of spark.memory.storageFraction in spark 2.3.x [Marketing Mail] [Marketing Mail]

rit : > This is just a counter to show you the size of cached RDDs. If it is zero > means that no caching has occurred. Also, even storage memory is used for > computing the counter will show as zero. > > Iacovos > On 20/3/20 4:51 μ.μ., Michel Sumbul wrote: > > Hi, > >

Re: Exact meaning of spark.memory.storageFraction in spark 2.3.x [Marketing Mail]

Hi, Thanks for the very quick reply! If I see the metrics "storage memory", always at 0, does that mean that the memory is neither used for caching or computing? Thanks, Michel Garanti s