Re: Save state on a CoGroupFunction and recover it after a failure

2021-06-16 Thread Felipe Gutierrez
I don't understand how I can save the state of a window on the RichCoGroupFunction if the events arrive on the RichCoGroupFunction.coCgroup only when the window closes. Then, upon a failure I will not recover events that were on the window. This is why I think the approach to this problem is to use

Re: Save state on a CoGroupFunction and recover it after a failure

2021-06-16 Thread Felipe Gutierrez
Hi Robert, 1 - I am using Kafka010 as data source. 2 - No, I am not using any kind of ListState. That I think it must be used 3 - Good. I am going to use CheckpointedFunction. Just a follow-up question. I was reimplementing it using CoProcessFunction to save the state and trigger the window. So,

Re: Save state on a CoGroupFunction and recover it after a failure

2021-06-16 Thread Robert Metzger
Hi Felipe, Which data source are you using? > Then, in the MyCoGroupFunction there are only events of stream02 Are you storing events in your state? > Is this the case where I have to use RichCoGroupFunction and save the state by implementing the CheckpointedFunction? If you want your state to

Save state on a CoGroupFunction and recover it after a failure

2021-06-15 Thread Felipe Gutierrez
Hi, I have a problem on my stream pipeline where the events on a CoGroupFunction are not restored after the application crashes. The application is like this: stream01.coGroup(stream02) .where(...).equalTo(...) .window(TumblingEventTimeWindows.of(1 minute)) .apply(new MyCoGroupFunction()) .proces