Hi Dave, As part of driver pod bringup, a configmap is created using all the spark configuration parameters (with name spark.properties) and mounted to /opt/spark/conf. So all the other files present in /opt/spark/conf will be overwritten. Same is happening with the log4j.properties in this case. You could try to build the container by placing the log4j.properties at some other location and set the same in spark.driver.extraJavaOptions
Thanks and Regards, Abhishek From: Dave Jaffe <dja...@vmware.com.INVALID> Sent: Tuesday, June 11, 2019 6:45 AM To: user@spark.apache.org Subject: Spark on Kubernetes - log4j.properties not read I am using Spark on Kubernetes from Spark 2.4.3. I have created a log4j.properties file in my local spark/conf directory and modified it so that the console (or, in the case of Kubernetes, the log) only shows warnings and higher (log4j.rootCategory=WARN, console). I then added the command COPY conf /opt/spark/conf to /root/spark/kubernetes/dockerfiles/spark/Dockerfile and built a new container. However, when I run that under Kubernetes, the program runs successfully but /opt/spark/conf/log4j.properties is not used (I still see the INFO lines when I run kubectl logs <driver pod>). I have tried other things such as explicitly adding a –properties-file to my spark-submit command and even --conf spark.driver.extraJavaOptions=-Dlog4j.configuration=file:///opt/spark/conf/log4j.properties My log4j.properties file is never seen. How do I customize log4j.properties with Kubernetes? Thanks, Dave Jaffe
