Thanks for your reply. "periodic reading" means reading all logs in a given time interval. For example, my logs is daily divided, I can get all logs of yesterday through a parameter like '2018-07-29/2018-07-30'.
TM which provides a web service to display information will lessen the burden of jobmanager, especially when there are many taskManagers in the flink cluster. 2018年7月30日星期一，vino yang <yanghua1...@gmail.com> 写道： > Hi Xinyu, > > This is indeed a problem. Especially when the amount of logs is large, it > may even cause the UI to stall for a long time. The same is true for YRAN, > and there is really no good way to do it at the moment. > Thank you for your suggestion, do you mean "periodic reading" refers to > full or incremental? If it is a full reading, I personally do not recommend > this, which will increase the burden on JM and client, and it is > appropriate to manually trigger it by the user. Maybe we can consider > loading only a few logs at a time, such as incremental reads, paged > displays, and so on. > Regarding the second question, Flink did this because its TM does not > provide any web services to display information, and the Web UI is > currently bundled with JM. > > Thanks, vino. > > 2018-07-30 16:33 GMT+08:00 Xinyu Zhang <wsz...@gmail.com>: > >> Hi all >> >> We use flink on yarn and flink version is 1.4. >> >> When a streaming job run for a long time, the webUI cannot show logs. >> This may be becasue the log size is too large. >> >> However, if we use the DailyRollingAppender to divide logs (granularity >> is `day`) in log4j.properties, we will never see the log of yesterday. >> >> Is there any ideas can make read logs easier? >> >> Maybe, we should add an interface that support for reading log by time >> interval. Besides, when we get the taskmanager logs through webUI, >> jobmanager can redirect to a URL of the taskmanager, which users can get >> the logs directly (Just like MR task), other than downloading the logs from >> taskmanager and then sending logs to users. >> >> Thanks! >> >> Xinyu Zhang >> >> >> >
