Hi Pieter, you're right that it would be nice to record the metrics for a later analysis. However, at the moment this is not supported. You could use the REST interface to obtain the JSON representation of the shown data in the web interface. By doing this repeatedly and parsing the metric data you can store it.
But I agree that this is not very nice. Maybe you could open a JIRA ticket to add this feature. Cheers, Till On Tue, Jan 26, 2016 at 9:22 PM, Ritesh Kumar Singh < riteshoneinamill...@gmail.com> wrote: > I didn't know these stats were collected. Thanks for telling :) > In that case, it should definitely be a feature which can be enabled via > config files. > > *Ritesh Kumar Singh,* > *https://riteshtoday.wordpress.com/* <https://riteshtoday.wordpress.com/> > > On Tue, Jan 26, 2016 at 8:22 PM, Pieter Hameete <phame...@gmail.com> > wrote: > >> Hi Ritesh, >> >> thanks for the response! The metrics are already being gathered though, >> so I think it would be nice to have a configuration/option to log them >> somewhere. It doesnt have to be enabled by default, and I dont think it >> should degrade the performance very much. It looks like the metrics are >> currently sent with each heartbeat by default already. Your Web UI probably >> hangs because it has to update all the graphs on every heartbeat, when you >> have many task managers that will be heavy on your computer :-) >> >> - Pieter >> >> 2016-01-26 20:17 GMT+01:00 Ritesh Kumar Singh < >> riteshoneinamill...@gmail.com>: >> >>> Going by the list in the latest documentation >>> <https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/monitoring_rest_api.html> >>> for Flink 0.10.1 release, memory and cpu stats are not stored. Neither is >>> the time spent on garbage collection stored anywhere. >>> >>> In my opinion, trying to store these metrics will degrade the >>> performance of jobs too. And so its basically a trade off between >>> performance and computation cost. For me, the web ui hangs even for the >>> current set of parameters :( >>> >>> *Ritesh Kumar Singh,* >>> *https://riteshtoday.wordpress.com/* >>> <https://riteshtoday.wordpress.com/> >>> >>> On Tue, Jan 26, 2016 at 7:16 PM, Pieter Hameete <phame...@gmail.com> >>> wrote: >>> >>>> Hi people! >>>> >>>> A lot of metrics are gathered for each TaskManager every few seconds. >>>> The web UI shows nice graphs for some of these metrics too. >>>> >>>> I would like to make graphs of the memory and cpu usage, and the time >>>> spent on garbage collection for each job. Because of this I am wondering if >>>> the metrics are also stored somewhere, or if there is an option to enable >>>> storing the metrics per job. >>>> >>>> In the configuration documentation I could not find such an option. Is >>>> this possible in version 0.9.1 of Flink? If not: is it possible in Flink >>>> 0.10.1 or is it possible to request or develop such a feature? >>>> >>>> Thank you for your help and kind regards, >>>> >>>> Pieter >>>> >>>> >>>> >>> >> >
