Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

. I haven't seen a use-case but I can imagine this is a > way for people to achieve better utilization with low priority long-running > jobs. > > On Fri, Mar 30, 2018 at 3:06 PM Yinan Li wrote: > >> Yes, the PR allows you to set say 1.5. The New configuration property >

Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

r > *spark.kubernetes.executor.cores > *would be. Seeing that I wanted more than 1 and Yinan wants less, leaving > it at 1 night be best. > > > > Thanks, > > David > > > > *From: *Kimoon Kim > *Date: *Friday, March 30, 2018 at 4:28 PM > *To: *Yinan Li

Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

task.cpus. > > A relevant question is should Spark on Kubernetes really be opinionated on > how to set the cpu request and limit and even try to determine this > automatically? > > On Fri, Mar 30, 2018 at 11:40 AM, Kimoon Kim > wrote: > >> > Instead of requesting `[d

Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

> Instead of requesting `[driver,executor].memory`, we should just request `[driver,executor].memory + [driver,executor].memoryOverhead `. I think this case is a bit clearer than the CPU case, so I went ahead and filed an issue with more details a

Re: Toward an "API" for spark images used by the Kubernetes back-end

Thanks for starting this discussion. When I was troubleshooting Spark on K8s, I often faced a need to turn on debug messages on the driver and executor pods of my jobs, which would be possible if I somehow put the right log4j.properties file inside the pods. I know I can build custom Docker images

Re: SPIP: Spark on Kubernetes

+1 (non-binding) Thanks, Kimoon On Tue, Aug 15, 2017 at 9:19 AM, Sean Suchter wrote: > +1 (non-binding) > > > > -- > View this message in context: http://apache-spark- > developers-list.1001551.n3.nabble.com/SPIP-Spark-on- > Kubernetes-tp22147p22150.html > Sent from the Apache Spark Developers