Re: Implement watermark buffering with Process Function

2022-02-16 Thread David Anderson
I've done some work on this with Nico Kruber. In our benchmarking, the performance loss (from not being able to use the namespace) was roughly a factor of two, so it is significant. We prototyped an API extension that addresses this particular concern but without exposing the namespace directly, w

Re: Implement watermark buffering with Process Function

2022-02-16 Thread David Anderson
I'm afraid not. The DataStream window implementation uses internal APIs to manipulate the state backend namespace, which isn't possible to do with the public-facing API. And without this, you can't implement this as efficiently. David On Wed, Feb 16, 2022 at 12:04 PM Ruibin Xing wrote: > Hi, >

Implement watermark buffering with Process Function

2022-02-16 Thread Ruibin Xing
Hi, I'm trying to implement customized state logic with KeyedProcessFunction. But I'm not quite sure how to implement the correct watermark behavior when late data is involved. According to the answer on stackoverflow: https://stackoverflow.com/questions/59468154/how-to-sort-an-out-of-order-event