> This is the case. Yanfei is correct that the uploading happens during the checkpoint stage.
Before checkpointing, there is a snapshotting stage that synchronously takes RocksDB snapshot by forcing all memtables into sstables on disk, and hard-linked in a local temporary directory. In the checkpointing stage, RocksDB snapshot will be uploaded to OSS or DFS fully or incrementally according to strategy. Best, tison. Yanfei Lei <fredia...@gmail.com> 于2023年7月21日周五 11:01写道： > Hi Taher, > > > is flink using rocks-cloud too? Or flink is moving rocks > files after checkpoints? > > Flink isn't using rocks-cloud. flink would upload local rocks files to > s3/DFS during checkpointing. Here is the related code: > > - RocksNativeFullSnapshotStrategy: > > https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksNativeFullSnapshotStrategy.java#L186 > - RocksIncrementalSnapshotStrategy: > > https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksIncrementalSnapshotStrategy.java#L291 > > Taher Koitawala <taher...@gmail.com> 于2023年7月21日周五 10:38写道： > > > > Hi all, > > I know that Flink uses rocksdb to write checkpoints. While > > researching rocksdb i found that there is something called rocks-cloud > that > > writes data over to s3. I know flink also does checkpoints over rocksdb > to > > s3. Question is, is flink using rocks-cloud too? Or flink is moving rocks > > files after checkpoints? > > > > Please share the code reference where this is done in flink. > > > > > > Regards, > > Taher Koitawala > > > > -- > Best, > Yanfei >
