Hello Spark fans, I am unable to figure out how Spark figures out which logger to use. I know that Spark decides upon this at the time of initialization of the Spark Context. From Spark documentation it is clear that Spark uses log4j, and not slf4j, but I have been able to successfully get spark to use slf4j logger before.
I ask because recently I ran into the typical "sfl4j and log4j" loop detected error ( http://mailman.qos.ch/pipermail/slf4j-user/2014-March/001412.html) and realized that although I excluded the slf4j-over-log4j12 bridge from my application, it was appearing in the class path. This was probably due to the fact that I run my jars as below, and somewhere: java -cp <my_spark_appjar>:<spark_core>:<hadoop_core> <mainClassName> I finally resolved by giving up and letting spark use log4j like it was "Defaulting to", and eliminating the log4j-over-slf4j jar from my fat-jars. The question is, how does spark decided which logging class to use.. slf4j or log4j.. and how can we control that choice? How could I get updated about what logger spark decides to go with. That also brings me to my second question. If there is a logger used by the spark context, can the same one be used by workers in the application? if yes, how? Cheers Shivani -- Software Engineer Analytics Engineering Team@ Box Mountain View, CA
