take-it-out opened a new pull request, #20399:
URL: https://github.com/apache/flink/pull/20399
## What is the purpose of the change
When rocksdb statebackend is used and state.backend.incremental enabled,
flink will figure out newly created sst files generated by rocksdb during
checkpoint, and read all the states from rocksdb and write to files during
savepoint.
When state.backend.incremental disabled, flink will read all the states from
rocksdb and generate state files in checkpoint and savepoint. This makes sense
in savepoint, cause user can take a savepoint with rocksdb statebackend and
then restore it using another statebackend, but in checkpoint, deserialisation
and serialisation of state results in performance loss.
If the native rocksdb snapshot is introduced in full snapshot, theoretically
better performance can be achieved. At the same time, savepoint remains the
same as before.
## Brief change log
This pull request changes the full snapshot in non-incremental checkpointing
when rocksdb statebackend is used , sst files will be persisted rather than
serialised states for better performance. Savepoint remains the same as before.
## Verifying this change
This change is already covered by existing tests, such as
*FullSnapshotRocksDbTtlStateTest*.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
