If the mentioned conf is enabled, the rolling of the stderr should work. If
it is not, then there is probably some bug. Take a look at the Worker's
logs and see if there is any error about rolling of the Executor's stderr.
If there is a bug, then it needs to be fixed (maybe you can take a crack at
I think that's up to you. You can make it log wherever you want, and
have some control over how log4j names the rolled log files by
configuring its file-based rolling appender.
On Thu, Feb 26, 2015 at 10:05 AM, Jeffrey Jedele
wrote:
> So the summarize (I had a similar question):
> Spark's log4j p
So the summarize (I had a similar question):
Spark's log4j per default is configured to log to the console? Those
messages end up in the stderr files and the approach does not support
rolling?
If I configure log4j to log to files, how can I keep the folder structure?
Should I use relative paths an
These settings don't control what happens to stderr, right? stderr is
up to the process that invoked the driver to control. You may wish to
configure log4j to log to files instead.
On Wed, Nov 12, 2014 at 8:15 PM, Nguyen, Duc wrote:
> I've also tried setting the aforementioned properties using
>
I'm also facing the same issue.
I tried the configurations but it seems the executors spark's
log4j.properties seems to override the passed values, so you have to change
/etc/spark/conf/log4j.properties.
Let me know if any of you have managed to get this fixes programatically.
I am planning to u
I've also tried setting the aforementioned properties using
System.setProperty() as well as on the command line while submitting the
job using --conf key=value. All to no success. When I go to the Spark UI
and click on that particular streaming job and then the "Environment" tab,
I can see the prop
We are running spark streaming jobs (version 1.1.0). After a sufficient
amount of time, the stderr file grows until the disk is full at 100% and
crashes the cluster. I've read this
https://github.com/apache/spark/pull/895
and also read this
http://spark.apache.org/docs/latest/configuration.html#
