There is a limitation in RocksDB's JNI bridge that will cause applications to fail if list state exceeds 2GB. I am not aware of anyone working on this issue.
Seth. [1] https://github.com/facebook/rocksdb/issues/2383 On Wed, Apr 8, 2020 at 12:02 PM Aaron Levin <aaronle...@stripe.com> wrote: > Hello friendly Flink community! > > I'm curious if anyone has operational experience with jobs that store > ListState where occasionally, due to skew, some small number of lists > stored in ListState (stored in RocksDB) will have millions of elements. > Here are the stats: > > * millions of keys > * p95 size of list in ListState is ~2. > * some small number of keys (less than 100) may have lists whose size is > on the order of tens of thousands and up to millions. > * state is stored in RocksDB > > Are there any known issues or limitations with storing or fetching that > much list state out of RocksDB? I realize fetching from RocksDB and > deserializing will be costly when hitting a key with a list of a million > elements, but is there anything else we should consider? > > Thanks! > > Best, > > Aaron Levin >
