Hi Stephan, Thanks for starting this discussion. I am +1 for this change. In general, number of timer state keys can have the same order as number of main state keys. So if RocksDB is used for main state for scalability, it makes sense to have timers there as well unless timers are used for only very limited subset of keys which fits into memory.
Best, Andrey On Thu, Jan 16, 2020 at 4:27 PM Stephan Ewen <se...@apache.org> wrote: > Hi all! > > I would suggest a change of the current default for timers. A bit of > background: > > - Timers (for windows, process functions, etc.) are state that is > managed and checkpointed as well. > - When using the MemoryStateBackend and the FsStateBackend, timers are > kept on the JVM heap, like regular state. > - When using the RocksDBStateBackend, timers can be kept in RocksDB > (like other state) or on the JVM heap. The JVM heap is the default though! > > I find this a bit un-intuitive and would propose to change this to let the > RocksDBStateBackend store all state in RocksDB by default. > The rationale being that if there is a tradeoff (like here), safe and > scalable should be the default and unsafe performance be an explicit choice. > > This sentiment seems to be shared by various users as well, see > https://twitter.com/StephanEwen/status/1214590846168903680 and > https://twitter.com/StephanEwen/status/1214594273565388801 > We would of course keep the switch and mention in the performance tuning > section that this is an option. > > # RocksDB State Backend Timers on Heap > - Pro: faster > - Con: not memory safe, GC overhead, longer synchronous checkpoint time, > no incremental checkpoints > > # RocksDB State Backend Timers on in RocksDB > - Pro: safe and scalable, asynchronously and incrementally checkpointed > - Con: performance overhead. > > Please chime in and let me know what you think. > > Best, > Stephan > >
