Hi, I want to manage logging of containers when I run Spark through YARN. I checked there is a environment variable exposed to custom log4j.properties.
Setting SPARK_LOG4J_CONF to "/dir/log4j.properties" should ideally make containers use "/dir/log4j.properties" file for logging. This doesn't seem working. Also, there is a log4j.properties file present in the Spark assembly and I guess it is picking that only. Quick Google gave me this: https://issues.apache.org/jira/browse/SPARK-2007 Is this functnality broken/not working yet? Or I m mistaken somewhere? Thanks. Archit.
