Re: Spark on Kubernetes

Hi, In k8s the driver is responsible for executor creation. The likelihood of your problem is that Insufficient memory allocated for executors in the K8s cluster. Even with dynamic allocation, k8s won't schedule executor pods if there is not enough free memory to fulfill their resource requests.

Re: [Spark on Kubernetes]: Seeking Guidance on Handling Persistent Executor Failures

Thanks for your kind words Sri Well it is true that as yet spark on kubernetes is not on-par with spark on YARN in maturity and essentially spark on kubernetes is still work in progress.* So in the first place IMO one needs to think why executors are failing. What causes this behaviour? Is it the

Re: [Spark on Kubernetes]: Seeking Guidance on Handling Persistent Executor Failures

Spark has supported the window-based executor failure-tracking mechanism for YARN for a long time, SPARK-41210[1][2] (included in 3.5.0) extended this feature to K8s. [1] https://issues.apache.org/jira/browse/SPARK-41210 [2] https://github.com/apache/spark/pull/38732 Thanks, Cheng Pan > On Fe

Re: [Spark on Kubernetes]: Seeking Guidance on Handling Persistent Executor Failures

Dear Mich, Thank you for your detailed response and the suggested approach to handling retry logic. I appreciate you taking the time to outline the method of embedding custom retry mechanisms directly into the application code. While the solution of wrapping the main logic of the Spark job in a l

Re: [Spark on Kubernetes]: Seeking Guidance on Handling Persistent Executor Failures

Went through your issue with the code running on k8s When an executor of a Spark application fails, the system attempts to maintain the desired level of parallelism by automatically recreating a new executor to replace the failed one. While this behavior is beneficial for transient errors, ensurin

Re: [Spark on Kubernetes]: Seeking Guidance on Handling Persistent Executor Failures

Not that I am aware of any configuration parameter in Spark classic to limit executor creation. Because of fault tolerance Spark will try to recreate failed executors. Not really that familiar with the Spark operator for k8s. There may be something there. Have you considered custom monitoring and

Re: spark on kubernetes

Glad to hear it! On Sun, Oct 16, 2022 at 2:37 PM Mohammad Abdollahzade Arani < mamadazar...@gmail.com> wrote: > Hi Qian, > Thanks for the reply and I'm So sorry for the late reply. > I found the answer. My mistake was token conversion. I had to decode > base64 the service accounts token and cert

Re: spark on kubernetes

Hi Mohammad Did you try this command? ./bin/spark-submit \ --master k8s://https://vm13:6443 \ --class com.example.WordCounter \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=default \ --conf spark.kubernetes.container.image=private-docker-registery/spark/spark:3.2.1-3

Re: Spark on Kubernetes scheduler variety

Splendid. Please invite me to the next meeting mich.talebza...@gmail.com Timezone London, UK *GMT+1* Thanks, view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or des

Re: Spark on Kubernetes scheduler variety

Hi Y'all, We had an initial meeting which went well, got some more context around Volcano and its near-term roadmap. Talked about the impact around scheduler deadlocking and some ways that we could potentially improve integration from the Spark side and Volcano sides respectively. I'm going to sta

Re: Spark on Kubernetes scheduler variety

Hi Holden, Thank you for your points. I guess coming from a corporate world I had an oversight on how an open source project like Spark does leverage resources and interest :). As @KlausMa kindly volunteered it would be good to hear scheduling ideas on Spark on Kubernetes and of course as I am su

Re: Spark on Kubernetes scheduler variety

That's awesome, I'm just starting to get context around Volcano but maybe we can schedule an initial meeting for all of us interested in pursuing this to get on the same page. On Wed, Jun 23, 2021 at 6:54 PM Klaus Ma wrote: > Hi team, > > I'm kube-batch/Volcano founder, and I'm excited to hear t

Re: Spark on Kubernetes scheduler variety

You can always chain aggregations by chaining multiple Structured Streaming jobs. It’s not a showstopper. Getting Spark on Kubernetes is important for organizations that want to pursue a multi-cloud strategy From: Mich Talebzadeh Date: Wednesday, June 23, 2021 at 11:27 AM To: "user @spark" Cc

Re: Spark on Kubernetes scheduler variety

Thanks Klaus! I am interested in more details. On Wed, Jun 23, 2021 at 6:54 PM Klaus Ma wrote: > Hi team, > > I'm kube-batch/Volcano founder, and I'm excited to hear that the spark > community also has such requirements :) > > Volcano provides several features for batch workload, e.g. fair-share

Re: Spark on Kubernetes scheduler variety

Thanks Klaus. That will be great. It will also be intuitive if you elaborate the need for this feature in line with the limitation of the current batch workload. Regards, Mich view my Linkedin profile *Disclaimer:* Use it at you

Re: Spark on Kubernetes scheduler variety

Hi team, I'm kube-batch/Volcano founder, and I'm excited to hear that the spark community also has such requirements :) Volcano provides several features for batch workload, e.g. fair-share, queue, reservation, preemption/reclaim and so on. It has been used in several product environments with Sp

Re: Spark on Kubernetes scheduler variety

Please allow me to be diverse and express a different point of view on this roadmap. I believe from a technical point of view spending time and effort plus talent on batch scheduling on Kubernetes could be rewarding. However, if I may say I doubt whether such an approach and the so-called democra

RE: Spark on Kubernetes | 3.0.1 | Shared Volume or NFS

Ok! Thanks for all guidance :-) Regards Ranju From: Mich Talebzadeh Sent: Thursday, March 11, 2021 11:07 PM To: Ranju Jain Cc: user@spark.apache.org Subject: Re: Spark on Kubernetes | 3.0.1 | Shared Volume or NFS I don't have any specific reference. However, you can do a Google search.

Re: Spark on Kubernetes | 3.0.1 | Shared Volume or NFS

> > > > Do you have any reference or links where I can check out the Shared > Volumes ? > > > > Regards > > Ranju > > > > *From:* Mich Talebzadeh > *Sent:* Thursday, March 11, 2021 5:38 PM > *Cc:* user@spark.apache.org > *Subject:* Re: Spar

RE: Spark on Kubernetes | 3.0.1 | Shared Volume or NFS

: user@spark.apache.org Subject: Re: Spark on Kubernetes | 3.0.1 | Shared Volume or NFS Well your mileage varies so to speak. The only way to find out is setting an NFS mount and testing it. The performance will depend on the mounted file system and the amount of cache it has. File cache is

Re: Spark on Kubernetes | 3.0.1 | Shared Volume or NFS

, March 11, 2021 5:22 PM > *To:* Ranju Jain > *Cc:* user@spark.apache.org > *Subject:* Re: Spark on Kubernetes | 3.0.1 | Shared Volume or NFS > > > > Ok this is on Google Cloud correct? > > > > > > > > > LinkedIn >

RE: Spark on Kubernetes | 3.0.1 | Shared Volume or NFS

the other sides [drawback]. Regards Ranju From: Mich Talebzadeh Sent: Thursday, March 11, 2021 5:22 PM To: Ranju Jain Cc: user@spark.apache.org Subject: Re: Spark on Kubernetes | 3.0.1 | Shared Volume or NFS Ok this is on Google Cloud correct? LinkedIn https://www.linkedin.com/profile

Re: Spark on Kubernetes | 3.0.1 | Shared Volume or NFS

Ok this is on Google Cloud correct? LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * *Disclaimer:* Use it at your own risk. Any and all responsibility for any los

RE: Spark on Kubernetes : unable to write files to HDFS

Everything is working fine now 🙂 Thanks again Loïc De : German Schiavon Envoyé : mercredi 16 décembre 2020 19:23 À : Loic DESCOTTE Cc : user@spark.apache.org Objet : Re: Spark on Kubernetes : unable to write files to HDFS We all been there! no reason to be

Re: Spark on Kubernetes : unable to write files to HDFS

6 décembre 2020 18:01 > *À :* Loic DESCOTTE > *Cc :* user@spark.apache.org > *Objet :* Re: Spark on Kubernetes : unable to write files to HDFS > > Hi, > > seems that you have a typo no? > > Exception in thread "main" java.io.IOException: No FileSystem for scheme

RE: Spark on Kubernetes : unable to write files to HDFS

Oh thank you you're right!! I feel shameful ?? De : German Schiavon Envoyé : mercredi 16 décembre 2020 18:01 À : Loic DESCOTTE Cc : user@spark.apache.org Objet : Re: Spark on Kubernetes : unable to write files to HDFS Hi, seems that you have a ty

Re: Spark on Kubernetes : unable to write files to HDFS

t; .appName("Hello Spark 7") > .config("fs.hdfs.impl", classOf[org.apache.hadoop.hdfs. > DistributedFileSystem].getName) > .getOrCreate() > > > But still the same error... > > -------------- > *De :* Sean Ow

RE: Spark on Kubernetes : unable to write files to HDFS

hdfs.impl", classOf[org.apache.hadoop.hdfs.DistributedFileSystem].getName) .getOrCreate() But still the same error... De : Sean Owen Envoyé : mercredi 16 décembre 2020 14:27 À : Loic DESCOTTE Objet : Re: Spark on Kubernetes : unable to write files to HDFS

Re: Spark on kubernetes : missing spark.kubernetes.driver.request.cores parameter ?

I am actually answering myself as I have check on master 3.x branch, and there is this feature ! https://issues.apache.org/jira/browse/SPARK-27754 So my understanding was correct. -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ -

RE: Spark on Kubernetes - log4j.properties not read

That did the trick, Abhishek! Thanks for the explanation, that answered a lot of questions I had. Dave -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.o

RE: Spark on Kubernetes - log4j.properties not read

Hi Dave, As part of driver pod bringup, a configmap is created using all the spark configuration parameters (with name spark.properties) and mounted to /opt/spark/conf. So all the other files present in /opt/spark/conf will be overwritten. Same is happening with the log4j.properties in this cas

Re: Spark on Kubernetes: Kubernetes killing executors because of overallocation of memory

Hi there, You may want to look at setting the memory overhead settings higher. Spark will then start containers with a higher memory limit (spark.executor.memory + spark.executor.memoryOverhead, to be exact) while the heap is still locked to spark.executor.memory. There’s some memory used by

Re: Spark on Kubernetes (minikube) 2.3 fails with class not found exception

This is the problem: > :/opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar;/opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar Seems like some code is confusing things when mixing OSes. It's using the Windows separator when building a command line ti be run on a Linux host. On Tue, Apr 1

Re: Spark on Kubernetes (minikube) 2.3 fails with class not found exception

Previous example was bad paste( I tried a lot of variants, so sorry for wrong paste ) PS C:\WINDOWS\system32> spark-submit --master k8s://https://ip:8443 --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=1 --executor-memory 1G --conf sp

Re: Spark on Kubernetes (minikube) 2.3 fails with class not found exception

The example jar path should be local:///opt/spark/examples/*jars* /spark-examples_2.11-2.3.0.jar. On Tue, Apr 10, 2018 at 1:34 AM, Dmitry wrote: > Hello spent a lot of time to find what I did wrong , but not found. > I have a minikube WIndows based cluster ( Hyper V as hypervisor ) and try > to