Hi Deepthi, 1. Have you had a look at flink-benchmarks [1]? I haven't used it but it might be helpful. 2. Unfortunately, Flink doesn't provide metrics like that. But you might want to follow FLINK-21736 [2] for future developments. 3. Is there anything specific you are looking for? Unfortunately, I don't know any blogs for a more detailed summary. If you plan to look into the code CheckpointCoordinator [3] might be a starting point. Alternatively, something like MetadataV2V3SerializerBase [4] offers insights into how the checkpoints' metadata is serialized.
Best, Matthias [1] https://github.com/apache/flink-benchmarks [2] https://issues.apache.org/jira/browse/FLINK-21736 [3] https://github.com/apache/flink/blob/11550edbd4e1874634ec441bde4fe4952fc1ec4e/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L1493 [4] https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83 On Tue, Mar 30, 2021 at 8:37 PM deepthi Sridharan < deepthi.sridha...@gmail.com> wrote: > Hi, > > I am trying to set up some benchmarking with a couple of IO options for > saving checkpoints and have a couple of questions : > > 1. Does flink come with any IO benchmarking tools? I couldn't find any. I > was hoping to use those to derive some insights about the storage > performance and extrapolate it for the checkpoint use case. > > 2. Are there any metrics pertaining to restore from checkpoints? The only > metric I can find is the last restore time, but neither the time it took to > read the checkpoints, nor the time it took to restore the operator/task > states seem to be covered. I am using RocksDB, but couldn't find any > metrics relating to how much time it took to restore the state backend from > rocksdb either. > > 3. I am trying to find documentation on how the states are serialized into > the checkpoint files from multiple operators and tasks to tailor the > testing use case, but can't seem to find any. Are there any bogs that go > into this detail or would reading the code be the only option? > > -- > Thanks, > Deepthi >
