Hello,I'm trying to use a custom log4j appender, with things specified in a
log4j.properties file. Very little seems to work in this regard. Here's what
I've tried:
1. Fat jar with logging dependencies included. log4j.properties in fat jar.
Spark doesn't pick up the properties file, so uses its defaults.
2. Fat jar with logging dependencies included. Submitted the properties file
like this:
spark-submit .... --files ./log4j.properties ........ --conf
'spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties'
fat.jarJust uses defaults. Doesn't use the submitted file.
3. Fat jar with logging dependencies included. Submitted the properties file
like this:
spark-submit .... --files ./log4j.properties ........ --conf
'spark.executor.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties'
fat.jarCrashes. Says log4j.properties file not found.
4. Fat jar with logging dependencies included. Submitted the properties file
like this:
spark-submit .... --files ./log4j.properties ........ --conf
'spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/localfolder/log4j.properties'
fat.jarCrashes. Says it could not find the appender stuff (JSonEventLayoutV1).
That's something that's provided in the fat jar.
5. Fat jar with logging dependencies included. Submitted the properties file
and logging jars like this: spark-submit .... --files ./log4j.properties
--jars 'comma,separated,list' ........ --conf
'spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/localfolder/log4j.properties'
fat.jarCrashes. Says it could not find the appender stuff (JSonEventLayoutV1).
That's something that's provided in the fat jar.
It seems only including the jars in extra class path is working. There seems to
be a somewhat related issue here:
https://issues.apache.org/jira/browse/SPARK-9826 , but that seems to have been
resolved in 1.4.1. I'm using 1.5.2. Is this a regression? Or am I missing
something?
Any help will be appreciated.
Thanks,Ashic.
