Yes each application can use its own log4j.properties but I am not sure how
to configure log4j so that the driver and executor write to file. This is
because if we set the "spark.executor.extraJavaOptions" it will read from a
file and that is not what I need.
How do I configure log4j from the app s
Hi,
I'm not completely sure about this either, but this is what we are doing
currently:
Configure your logging to write to STDOUT, not to a file explicitely. Spark
will capture stdour and stderr and separate the messages into a app/driver
folder structure in the configured worker directory.
We the
Are these jobs the same jobs, just run by different users or, different
jobs ?
If the latter, can each application use its own log4j.properties ?
Cheers
On Fri, Mar 20, 2015 at 1:43 PM, Udit Mehta wrote:
> Hi,
>
> We have spark setup such that there are various users running multiple
> jobs at
Hi,
We have spark setup such that there are various users running multiple jobs
at the same time. Currently all the logs go to 1 file specified in the
log4j.properties.
Is it possible to configure log4j in spark for per app/user logging instead
of sending all logs to 1 file mentioned in the log4j.
