Zhong Chunlai created SPARK-19077:
-------------------------------------
Summary: no way to prevent spark streaming's eventLog inprogress
file becoming too big?
Key: SPARK-19077
URL: https://issues.apache.org/jira/browse/SPARK-19077
Project: Spark
Issue Type: Question
Affects Versions: 2.0.0
Reporter: Zhong Chunlai
My spark streaming job generates a large eventLog inprogess file after some
days, which is more than 200G. And it is becoming even bigger and bigger as
spark streaming continues running. I think it is not possible for history
server is to load this eventLog file as it ends. is there any way to make it
smaller, or make it a rolling log?
Also I read spark code, I find eventLog is not log4j log, it can't set log
level!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
