Hi Fabian and Till, thanks for the tips i'll see if I can work with the REST interface for now. I'll make a JIRA ticket as well. I might even be able to develop this feature but I wont have time to do that in the coming 2 months. It would be nice to be able to make a first contribution though. Keep up the good work :-)
- Pieter 2016-01-27 11:04 GMT+01:00 Till Rohrmann <trohrm...@apache.org>: > Hi Pieter, > > you're right that it would be nice to record the metrics for a later > analysis. However, at the moment this is not supported. You could use the > REST interface to obtain the JSON representation of the shown data in the > web interface. By doing this repeatedly and parsing the metric data you can > store it. > > But I agree that this is not very nice. Maybe you could open a JIRA ticket > to add this feature. > > Cheers, > Till > > On Tue, Jan 26, 2016 at 9:22 PM, Ritesh Kumar Singh < > riteshoneinamill...@gmail.com> wrote: > >> I didn't know these stats were collected. Thanks for telling :) >> In that case, it should definitely be a feature which can be enabled via >> config files. >> >> *Ritesh Kumar Singh,* >> *https://riteshtoday.wordpress.com/* <https://riteshtoday.wordpress.com/> >> >> On Tue, Jan 26, 2016 at 8:22 PM, Pieter Hameete <phame...@gmail.com> >> wrote: >> >>> Hi Ritesh, >>> >>> thanks for the response! The metrics are already being gathered though, >>> so I think it would be nice to have a configuration/option to log them >>> somewhere. It doesnt have to be enabled by default, and I dont think it >>> should degrade the performance very much. It looks like the metrics are >>> currently sent with each heartbeat by default already. Your Web UI probably >>> hangs because it has to update all the graphs on every heartbeat, when you >>> have many task managers that will be heavy on your computer :-) >>> >>> - Pieter >>> >>> 2016-01-26 20:17 GMT+01:00 Ritesh Kumar Singh < >>> riteshoneinamill...@gmail.com>: >>> >>>> Going by the list in the latest documentation >>>> <https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/monitoring_rest_api.html> >>>> for Flink 0.10.1 release, memory and cpu stats are not stored. Neither is >>>> the time spent on garbage collection stored anywhere. >>>> >>>> In my opinion, trying to store these metrics will degrade the >>>> performance of jobs too. And so its basically a trade off between >>>> performance and computation cost. For me, the web ui hangs even for the >>>> current set of parameters :( >>>> >>>> *Ritesh Kumar Singh,* >>>> *https://riteshtoday.wordpress.com/* >>>> <https://riteshtoday.wordpress.com/> >>>> >>>> On Tue, Jan 26, 2016 at 7:16 PM, Pieter Hameete <phame...@gmail.com> >>>> wrote: >>>> >>>>> Hi people! >>>>> >>>>> A lot of metrics are gathered for each TaskManager every few seconds. >>>>> The web UI shows nice graphs for some of these metrics too. >>>>> >>>>> I would like to make graphs of the memory and cpu usage, and the time >>>>> spent on garbage collection for each job. Because of this I am wondering >>>>> if >>>>> the metrics are also stored somewhere, or if there is an option to enable >>>>> storing the metrics per job. >>>>> >>>>> In the configuration documentation I could not find such an option. Is >>>>> this possible in version 0.9.1 of Flink? If not: is it possible in Flink >>>>> 0.10.1 or is it possible to request or develop such a feature? >>>>> >>>>> Thank you for your help and kind regards, >>>>> >>>>> Pieter >>>>> >>>>> >>>>> >>>> >>> >> >
