Hello Alex
Thanks for the link. Yes creating a singleton object for logging outside
the code that gets executed on the workers definitely works. The problem
that i am facing though is related to configuration of the logger. I don't
see any log messages in the worker logs of the application.
a) wh
Hi,
I had the same problem with pyspark. Here's how I resolved it:
What I've found in python (not sure about scala) is that if the function
being serialized was written in the same python module as the main
function, then logging fails. If the serialized function is in a separate
module, then
Hello Spark fans,
I am trying to log messages from my spark application. When the main()
function attempts to log, using log.info() it works great, but when I try
the same command from the code that probably runs on the worker, I
initially got an serialization error. To solve that, I created a new
