Re: Task Manager metrics per job on Flink 0.9.1

I didn't know these stats were collected. Thanks for telling :) In that case, it should definitely be a feature which can be enabled via config files. *Ritesh Kumar Singh,* *https://riteshtoday.wordpress.com/* On Tue, Jan 26, 2016 at 8:22 PM, Pieter Hameete w

Re: Task Manager metrics per job on Flink 0.9.1

Hi Ritesh, thanks for the response! The metrics are already being gathered though, so I think it would be nice to have a configuration/option to log them somewhere. It doesnt have to be enabled by default, and I dont think it should degrade the performance very much. It looks like the metrics are

Re: Task Manager metrics per job on Flink 0.9.1

Going by the list in the latest documentation for Flink 0.10.1 release, memory and cpu stats are not stored. Neither is the time spent on garbage collection stored anywhere. In my opinion, trying to s

Streaming left outer join

Hello, I'm trying to implement a left outer join of two Kafka streams within a sliding window. So far I have the following code: foos .coGroup(bars) .where(_.baz).equalTo(_.baz) .window(SlidingTimeWindows.of(Time.of(1, TimeUnit.MINUTES), Time.of(1, TimeUnit.SECONDS))) .apply((fs: Iterator

Task Manager metrics per job on Flink 0.9.1

Hi people! A lot of metrics are gathered for each TaskManager every few seconds. The web UI shows nice graphs for some of these metrics too. I would like to make graphs of the memory and cpu usage, and the time spent on garbage collection for each job. Because of this I am wondering if the metric

Re: rowmatrix equivalent

Hi Till, maybe I will do that :) If I have some other questions I will let you know! Best regards, Lydia > Am 24.01.2016 um 17:33 schrieb Till Rohrmann : > > Hi Lydia, > > Flink does not come with a distributed matrix implementation as Spark does it > with the RowMatrix, yet. However, you c

RE: continous time triger

Hi, Thank you for sharing your experience and also to Till for the advice. What I would like to do is to be able to fire the window potentially multiple times, even if an event did not arrive. I will look more about how dealing with the processing time could help in this Dr. Radu Tudoran Resear

Re: continous time triger

For what it's worth, we have a trigger that fires once a day for a recurring calculation. When an element comes in, we set the trigger context's processing time timer to the exact millisecond of the desired time. The predefined triggers were useful to look at to achieve this: https://github.com/apa

Re: maxtime / watermark for GlobaWindow

Hi Radu, If I’m not mistaken, then it’s not possible with the current GlobalWindow implementation. However, you could extend the GlobalWindow which adds a new field into which the timestamp of the triggering element is stored. This field can then be read from within the WindowFunction to retrieve

Re: Redeployements and state

Hey Niels! Stephan gave a very good summary of the current state of things. What do you think of the outlined stop with savepoint method? Regarding the broken links: I’ve fixed various broken links in the master docs yesterday. If you encounter something again, feel free to post it to the ML or

Re: continous time triger

Hi Radu, you can register processing and event time time triggers using the TriggerContext which is given to the onElement, onProcessingTime and onEventTime methods of Trigger. In case you register a processing time timer, the onProcessingTime method will be called once the system clock has passed