Thanks guys! This was really helpful On Fri, 21 Jul 2023 at 8:49 AM, tison <wander4...@gmail.com> wrote:
> > This is the case. > > Yanfei is correct that the uploading happens during the checkpoint stage. > > Before checkpointing, there is a snapshotting stage that synchronously > takes RocksDB snapshot by forcing all memtables into sstables on disk, and > hard-linked in a local temporary directory. > > In the checkpointing stage, RocksDB snapshot will be uploaded to OSS or DFS > fully or incrementally according to strategy. > > Best, > tison. > > > Yanfei Lei <fredia...@gmail.com> 于2023年7月21日周五 11:01写道： > > > Hi Taher, > > > > > is flink using rocks-cloud too? Or flink is moving rocks > > files after checkpoints? > > > > Flink isn't using rocks-cloud. flink would upload local rocks files to > > s3/DFS during checkpointing. Here is the related code: > > > > - RocksNativeFullSnapshotStrategy: > > > > > https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksNativeFullSnapshotStrategy.java#L186 > > - RocksIncrementalSnapshotStrategy: > > > > > https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksIncrementalSnapshotStrategy.java#L291 > > > > Taher Koitawala <taher...@gmail.com> 于2023年7月21日周五 10:38写道： > > > > > > Hi all, > > > I know that Flink uses rocksdb to write checkpoints. While > > > researching rocksdb i found that there is something called rocks-cloud > > that > > > writes data over to s3. I know flink also does checkpoints over rocksdb > > to > > > s3. Question is, is flink using rocks-cloud too? Or flink is moving > rocks > > > files after checkpoints? > > > > > > Please share the code reference where this is done in flink. > > > > > > > > > Regards, > > > Taher Koitawala > > > > > > > > -- > > Best, > > Yanfei > > >
