Re: RocksDB savepoint recovery performance improvements

2020-05-26 Thread Joey Pereira
as well as getting some RocksDB benchmarks. On Mon, May 18, 2020 at 3:46 PM Joey Pereira wrote: > Thanks Yun for highlighting this, it's very helpful! I'll give it a go > with that in mind. > > We have already begun using checkpoints for recovery. Having these > improvemen

Re: RocksDB savepoint recovery performance improvements

2020-05-18 Thread Joey Pereira
hotStrategy.java#L308 > > > Best > Yun Tang > -- > *From:* Joey Pereira > *Sent:* Tuesday, May 19, 2020 2:27 > *To:* user@flink.apache.org > *Cc:* Mike Mintz ; Shahid Chohan ; > Aaron Levin > *Subject:* RocksDB savepoint recovery perfor

RocksDB savepoint recovery performance improvements

2020-05-18 Thread Joey Pereira
Hey, While running a Flink application with a large-state, savepoint recovery has been a painful part of operating the application because recovery time can be several hours. During some profiling that chohan (cc'd) had done, a red flag stood out — savepoint recovery consisted mostly of RocksDB Ge