Hi,
I am trying to change the log4j configuration for jobs submitted to a k8s
cluster (using submit)
The my-log4j.xml is uploaded using --files ,./my-log4j.xml and the file in
the working directory of the driver/exec pods.
I added D-flags using the extra java options (and tried many different
URIs, absolute, with and without file:.
--conf spark.driver.extraJavaOptions="-Dlog4j2.debug=false
-Dlog4j2.configurationFile=file:./my-log4j.xml" \
--conf spark.executor.extraJavaOptions="-Dlog4j2.debug=false
-Dlog4j2.configurationFile=file:./my-log4j.xml" \
When debugging i notice that log4j is not able to load my configuration
file. I see the following additional log entries:
ERROR StatusLogger Reconfiguration failed: No configuration found for
'4a87761d' at 'null' in 'null'
ERROR StatusLogger Reconfiguration failed: No configuration found for
'Default' at 'null' in 'null'
24/06/06 09:20:44 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where
applicable
Files
file:///tmp/mk2/spark-upload-36b1f43d-1878-4b06-be5e-e25b703f28d5/orbit-movements.csv
from
/tmp/mk2/spark-upload-36b1f43d-1878-4b06-be5e-e25b703f28d5/orbit-movements.csv
to /opt/spark/work-dir/orbit-movements.csv
Files
file:///tmp/mk2/spark-upload-cc211704-f481-4ebe-b6f0-5dbe66a7c639/my-log4j.xml
from /tmp/mk2/spark-upload-cc211704-f481-4ebe-b6f0-5dbe66a7c639/my-log4j.xml
to /opt/spark/work-dir/my-log4j.xml
Files file:///tmp/mk2/spark-upload-7970b482-7669-49aa-9f88-65191a83a18a/out.jar
from /tmp/mk2/spark-upload-7970b482-7669-49aa-9f88-65191a83a18a/out.jar
to /opt/spark/work-dir/out.jar
The lines starting with Files in the logs of the Driver process, makes me
wonder if the copying of files from my shared mount to the working
directory happens in that process and is not something that happens before
the java process launches. Is that assumption correct, as it would explain
why my log4j config files are not found at JVM launch.
If so, what is the recommended way to change the logging config *per job*
when running spark in k8s (i am not using a custom container image, so
can’t place it in there)
tx.,
