Re: Combining multiple stages into a multi-stage processing pipeline

2024-04-07 Thread Mark Petronic
Thank you Yunfeng. Your comments gave me some insights to explore how to use consecutive windows. So, I coded up a version that looks like this and works well for me: KafkaSource => Keyby => TumblingWindows => ProcessWindowFn => WindowAll => ProcessWindowFn => (Here I will repeated keyed and windo

Re: Combining multiple stages into a multi-stage processing pipeline

2024-04-06 Thread Yunfeng Zhou
Hi Mark, IMHO, your design of the Flink application is generally feasible. In Flink ML, I have once met a similar design in ChiSqTest operator, where the input data is first aggregated to generate some results and then broadcast and connected with other result streams from the same input afterward