Hi, OpenInx: Yes, I've read that pr. But if we want to maintain the out of memory map, I lean towards using flink's operator state, which has already provided high level api for us, including the type serialization, configuration, etc. It would be easier to maintain by reusing what flink has provided. Also, it would be easy to switch between heap states if the user is sure that their state is not large and wants high performance. What do you think?
On Mon, Dec 11, 2023 at 11:15 AM OpenInx <open...@gmail.com> wrote: > Just provided a little context: there was a stale PR which was trying to > maintain the insertedRowMap into RocksDB.. > > On Sat, Dec 9, 2023 at 1:52 AM Ryan Blue <b...@tabular.io> wrote: > >> Thanks, Renjie! >> >> The option to use Flink's state tracking system seems like a good idea to >> me. >> >> On Thu, Dec 7, 2023 at 8:19 PM Renjie Liu <liurenjie2...@gmail.com> >> wrote: >> >>> Hi: >>> I want to raise a discussion about controlling flink's upsert sink's >>> memory usage: >>> >>> https://toys-flash-4hl.craft.me/3VHrdWbV30QMk6 >>> >>> Welcome to comment and share your thoughts. >>> >> >> >> -- >> Ryan Blue >> Tabular >> >
