Hi Vishal, FIFO compaction with RocksDB can result in data loss as it discards the oldest SST file by a size-based trigger or based on TTL (an internal RocksDB option, irrelevant to the TTL setting in Flink). So use with cause.
I'm not sure why you observed SST file compactions, did you manually enabled allow compaction option for FIFO? Best, Zhanghao Chen ________________________________ From: Alexander Fedulov <alexan...@ververica.com> Sent: Tuesday, July 5, 2022 5:49 To: Vishal Surana <vis...@moengage.com> Cc: user <user@flink.apache.org> Subject: Re: Can FIFO compaction with RocksDB result in data loss? Hi Vishal, I am not sure I get what you mean by the question #2: >2. SST files get created each time a checkpoint is triggered. At this point, >does the data for a given key get merged in case the initial data was read >from an SST file while the update must have happened in memory? Could you maybe rephrase it? Upon the arrival of a checkpoint barrier, Flink flushes all memtables to SST files synchronously (so any state updates are shortly blocked). Compaction runs in background thread(s) and Flink deals with the outcome of RocksDB compaction post factum. For non-incremental checkpoints, everything is straightforward - all SST files are copied over. The interplay of the incremental checkpoints and compaction is described in this [1] blog post. [1] https://flink.apache.org/features/2018/01/30/incremental-checkpointing.html Best, Alexander Fedulov On Mon, Jul 4, 2022 at 4:25 PM Vishal Surana <vis...@moengage.com<mailto:vis...@moengage.com>> wrote: In my load tests, I've found FIFO compaction to offer the best performance as my job needs state only for so long. However, this particular statement in RocksDB documentation concerns me: "Since we never rewrite the key-value pair, we also don't ever apply the compaction filter on the keys." This is from - https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style I've observed that SST files are getting compacted into larger SST files until a configured threshold is reached. Thus I'm not sure what's going on anymore. My questions at this stage are: 1. If there's a value that I get from RocksDB and I decide to update this value back then will it work with FIFO compaction? 2. SST files get created each time a checkpoint is triggered. At this point, does the data for a given key get merged in case the initial data was read from an SST file while the update must have happened in memory? 3. If the answer to above is yes, then I suppose I can use FIFO compaction. However, then my question is whether the RocksDB documentation is wrong or whether Flink is doing something in addition to what RocksDB does. Thanks!
