Hello Spark fans, I am trying to log messages from my spark application. When the main() function attempts to log, using log.info() it works great, but when I try the same command from the code that probably runs on the worker, I initially got an serialization error. To solve that, I created a new logger in the code that operates on the data, which solved the serialization issue but now there is no output in the console or on the worker node logs. I don't see any application level log messages in the spark logs either. When I use println() instead, I do see console output being generated.
I tried the following and none of them works a) pass log4j.properties by using -Dlog4j.properties in my java command line initiation of the spark application b) setting the properties within the worker by calling log.addAppender(new ConsoleAppender) None of them work. What am i missing? Thanks, Shivani -- Software Engineer Analytics Engineering Team@ Box Mountain View, CA
