Hi Taher, > is flink using rocks-cloud too? Or flink is moving rocks files after checkpoints?
Flink isn't using rocks-cloud. flink would upload local rocks files to s3/DFS during checkpointing. Here is the related code: - RocksNativeFullSnapshotStrategy: https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksNativeFullSnapshotStrategy.java#L186 - RocksIncrementalSnapshotStrategy: https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksIncrementalSnapshotStrategy.java#L291 Taher Koitawala <taher...@gmail.com> 于2023年7月21日周五 10:38写道： > > Hi all, > I know that Flink uses rocksdb to write checkpoints. While > researching rocksdb i found that there is something called rocks-cloud that > writes data over to s3. I know flink also does checkpoints over rocksdb to > s3. Question is, is flink using rocks-cloud too? Or flink is moving rocks > files after checkpoints? > > Please share the code reference where this is done in flink. > > > Regards, > Taher Koitawala -- Best, Yanfei
