Re: Problem with ProcessFunction timeout feature

2017-06-23 Thread Álvaro Vilaplana García
Well, it sounds very reasonable to me! I will let you know how it goes. 2017-06-23 10:05 GMT+01:00 Stefan Richter : > Yes, exactly. The idea would be, that you operate in event time, but > combine it with processing time timers to trigger timeout detection. Could > that help for your case? > >

Re: Problem with ProcessFunction timeout feature

2017-06-23 Thread Stefan Richter
Yes, exactly. The idea would be, that you operate in event time, but combine it with processing time timers to trigger timeout detection. Could that help for your case? > Am 23.06.2017 um 10:55 schrieb Álvaro Vilaplana García > : > > Hi Stefan, > > You meant > > /** > * Registers a timer t

Re: Problem with ProcessFunction timeout feature

2017-06-23 Thread Álvaro Vilaplana García
Hi Stefan, You meant /** * Registers a timer to be fired when processing time passes the given time. * * Timers can internally be scoped to keys and/or windows. When you set a timer * in a keyed context, such as in an operation on * {@link org.apache.flink.streaming.api.datastream.KeyedStrea

Re: Problem with ProcessFunction timeout feature

2017-06-23 Thread Álvaro Vilaplana García
Hi Stefan, Thank you for your knowledge, very appreciated. According with the documentation: void registerEventTimeTimer(long time); -> 'Registers a timer to be fired when the event time watermark passes the given time.' Dont we have the same problem? We would need an event (that event does not

Re: Problem with ProcessFunction timeout feature

2017-06-23 Thread Stefan Richter
Hi, yes, I think you understood the basic concept of watermarks. Events are basically driving „the event time clock“, so it can only advance when you see events. I am not sure if I got the part about partitions correctly, but the watermark event time is a global thing. For example, if you have

Re: Problem with ProcessFunction timeout feature

2017-06-23 Thread Álvaro Vilaplana García
Hi Stefan, Thank you so much for your answer. Regarding the 'artificial events', our main problem is that we have no control at all in the devices. I have been reading more about event time and watermarks and what I understood is that when we use event times (device times) Flink does not know an

Re: Problem with ProcessFunction timeout feature

2017-06-22 Thread Stefan Richter
Hi, if I understand correctly, your problem is that event time does not progress in case you don’t receive events, so you cannot detect the timeout of devices. Would it make sense to have you source periodically send artificial events to advance the watermark in the absence of device events, wi

Problem with ProcessFunction timeout feature

2017-06-22 Thread Álvaro Vilaplana García
Hi, Please, can you help me with a problem? I summarise in the next points, I hope is enough clear to approach some help. a) We have devices, each with its own ID, which we don’t have control of b) These devices send messages, with an internally generated, non-synced (amongst other devices) tim