Thanks a lot for the help! Yu'an and Martijn. To share and confirm my understanding, the recipe using CURRENT_WATERMARK to get late data will return all data arriving later than the defined bounded out-of-orderness, without consideration of window closing time. In comparison, WindowedStream.sideOutputLateData(OutputTag) will only return data that arrives after its window is already closed.
On Thu, Mar 31, 2022 at 2:26 AM Martijn Visser <martijnvis...@apache.org> wrote: > Hi, > > The only thing you currently can do is filter out late data using the > CURRENT_WATERMARK function since Flink 1.14. There's a SQL Cookbook recipe > on this function which can be found at > https://github.com/ververica/flink-sql-cookbook/blob/main/other-builtin-functions/03_current_watermark/03_current_watermark.md > > Best regards, > > Martijn Visser > https://twitter.com/MartijnVisser82 > https://github.com/MartijnVisser > > > On Thu, 31 Mar 2022 at 11:00, yu'an huang <h.yuan...@gmail.com> wrote: > >> Hi, in my understanding, Flink only support to get late data by side >> output in data stream api currently. For Table API/SQL, unfortunately, late >> events will always be dropped. >> >> You can see this link as reference: >> https://stackoverflow.com/questions/60218235/using-event-time-with-lateness-in-flink-sql-windows >> >> >> >> > On 31 Mar 2022, at 5:38 AM, liuxiangcao <xiangcaohe...@gmail.com> >> wrote: >> > >> > Hi Flink community, >> > >> > In Flink DataStream Java API, user can get get data that was discarded >> as late using WindowedStream.sideOutputLateData(OutputTag) (see [1]). I'm >> wondering what is the best way for user to achieve this in Flink SQL? >> > >> > For background, we are providing pure sql deployment to our internal >> users which means user won't be using Flink Table API directly. They will >> write Flink SQL script, with Java only used for UDF. >> > >> > Would appreciate if any one here can share your insights or >> experiences. Thanks! >> > >> > [1] >> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/#getting-late-data-as-a-side-output >> > >> > >> > >> >> -- Best Wishes & Regards Xiangcao Liu
