Re: Incremental state with purging

2020-05-18 Thread Thomas Huang
state with purging Hi, Thanks for your suggestions! However, as I'm reading the docs for queryable state, it says that it can only be used for Processing time, and my windows are defined using event time. So, I guess this means I should use the KeyedProcessFunction. Could you maybe sugg

Re: Incremental state with purging

2020-05-18 Thread Annemarie Burger
Hi, Thanks for your suggestions! However, as I'm reading the docs for queryable state, it says that it can only be used for Processing time, and my windows are defined using event time. So, I guess this means I should use the KeyedProcessFunction. Could you maybe suggest a rough implementation fo

Re: Incremental state with purging

2020-05-15 Thread Congxian Qiu
last y seconds. > > > [1] > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#state-time-to-live-ttl > > Best > Yun Tang > -- > *From:* Annemarie Burger > *Sent:* Wednesday, May 13, 2020 2:46 > *To:* user@flink.apac

Re: Incremental state with purging

2020-05-13 Thread Yun Tang
seconds. [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#state-time-to-live-ttl Best Yun Tang From: Annemarie Burger Sent: Wednesday, May 13, 2020 2:46 To: user@flink.apache.org Subject: Incremental state with purging Hi, I

Incremental state with purging

2020-05-12 Thread Annemarie Burger
Hi, I'm trying to implement the most efficient way to incrementally put incoming DataStream elements in my (map)state, while removing old elements (older that x) from that same state. I then want to output the state every y seconds. I've looked into using the ProcessFunction with onTimer, or build