There's really no current solution to this. There's a brief discussion about it on SPARK-12140.
Here we recommend people disable event logs for streaming apps, as sub-optimal as that might be... On Fri, Dec 1, 2017 at 3:27 AM, kankalapti omkar naidu <omkarnaid...@gmail.com> wrote: > Dear spark developers, > > I did not find rolling policy for spark event logs,for long living streaming > applications the event log is becoming huge with in few hours. > The only way I found is fixing the existing event logging spark > listener(EventLoggingListener), > but I believe there should be already a solution tackling this problem > because it is a frequently occurring problem when ever somebody runs > streaming applications for long time. > > What would be the right way to fix this,whether any config settings are > available for this or writing our own scripts to cleanup the log directory > regularly based on time/size or we need to implement rolling policy feature? > > Regards > Omkar -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
