Also, I have YARN aggregated enabled. But because logs are gone locally, after the job is killed or failed. I don't have aggregated logs on HDFS.
On Tue, Aug 25, 2015 at 5:39 PM, Chen Song <chen.song...@gmail.com> wrote: > Thanks Navina. > > What I found is that containers logs were gone after days locally even > when the job is still running. > > Chen > > On Mon, Aug 3, 2015 at 6:06 PM, Navina Ramesh < > nram...@linkedin.com.invalid> wrote: > >> Hi Chen, >> You can set the yarn.nodemanage.delete.debug-delay-sec config in >> yarn-site.xml, in order to retain the logs after the job crashes. Refer >> >> https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml >> >> >> (This was intended for debug purpose). >> >> If you need to retain log for longer period of time for purposes other >> than >> debugging, you should look into yarn's log aggregation features. >> Alternatively, you can use Samza's StreamAppender to produce the log data >> to a stream. You can then publish it to ELK for further lookup. It really >> depends on your requirements. >> Please note that the StreamAppender has a bug in the master branch that is >> pending resolution in https://issues.apache.org/jira/browse/SAMZA-723 >> >> Cheers! >> Navina >> >> >> On Mon, Aug 3, 2015 at 2:26 PM, Chen Song <chen.song...@gmail.com> wrote: >> >> > Dump question. >> > >> > When running samza job on YARN, I found that container logs gone after >> > days. Is there a YARN config to keep the logs from being deleted for >> > long-lived streaming job? >> > >> > -- >> > Chen Song >> > >> >> >> >> -- >> Navina R. >> > > > > -- > Chen Song > > -- Chen Song
