Hi Gabriele, Quick answer: You can use the built-in window operators which have been integrated with state backends including RocksDB.
Thanks, Zakelly On Tue, Mar 5, 2024 at 10:33 AM Zhanghao Chen <zhanghao.c...@outlook.com> wrote: > Hi Gabriele, > > I'd recommend extending the existing window function whenever possible, as > Flink will automatically cover state management for you and no need to be > concerned with state backend details. Incremental aggregation for reduce > state size is also out of the box if your usage can be satisfied with the > reduce/aggregate function pattern, which is important for large windows. > > Best, > Zhanghao Chen > ------------------------------ > *From:* Gabriele Mencagli <gabriele.menca...@gmail.com> > *Sent:* Monday, March 4, 2024 19:38 > *To:* user@flink.apache.org <user@flink.apache.org> > *Subject:* Question about time-based operators with RocksDB backend > > > Dear Flink Community, > > I am using Flink with the DataStream API and operators implemented using > RichedFunctions. I know that Flink provides a set of window-based operators > with time-based semantics and tumbling/sliding windows. > > By reading the Flink documentation, I understand that there is the > possibility to change the memory backend utilized for storing the in-flight > state of the operators. For example, using RocksDB for this purpose to cope > with a larger-than-memory state. If I am not wrong, to transparently change > the backend (e.g., from in-memory to RocksDB) we have to use a proper API > to access the state. For example, the Keyed State API with different > abstractions such as ValueState<T>, ListState<T>, etc... as reported here > <https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/fault-tolerance/state/> > . > > My question is related to the utilization of time-based window operators > with the RocksDB backend. Suppose for example very large temporal windows > with a huge number of keys in the stream. I am wondering if there is a > possibility to use the built-in window operators of Flink (e.g., with an > AggregateFunction or a more generic ProcessWindowFunction as here > <https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/operators/windows/>) > transparently with RocksDB support as a state back-end, or if I have to > develop the window operator in a raw manner using the Keyed State API > (e.g., ListState, AggregateState) for this purpose by implementing the > underlying window logic manually in the code of RichedFunction of the > operator (e.g., a FlatMap). > Thanks for your support, > > -- > Gabriele Mencagli > >
