Re: Issue while building spark project

Thanks a lot Sean On Mon, Jul 18, 2022, 21:58 Sean Owen wrote: > Increase the stack size for the JVM when Maven / SBT run. The build sets > this but you may still need something like "-Xss4m" in your MAVEN_OPTS > > On Mon, Jul 18, 2022 at 11:18 AM rajat kumar > wrote: > >> Hello , >> >> Can any

Re: spark.executor.pyspark.memory not added to the executor resource request on Kubernetes

... using spark 3.2.1 From: Shay Elbaz Sent: Tuesday, July 19, 2022 1:26 PM To: user@spark.apache.org Cc: Jeffrey O'Donoghue Subject: [EXTERNAL] spark.executor.pyspark.memory not added to the executor resource request on Kubernetes ATTENTION: This email origin

spark.executor.pyspark.memory not added to the executor resource request on Kubernetes

Hi, We are trying tune executor memory on Kubernetes. Specifically, 8g for the jvm, 8g for the python process, and additional 500m overhead: --conf spark.executor.memory=8g --conf spark.executor.pyspark.memory=8g --conf spark.executor.memoryOverhead=500m According the docs, the executor pods sh