azagrebin commented on a change in pull request #6777: [FLINK-10461] [State
Backends, Checkpointing] Speed up download files when restore from DFS
URL: https://github.com/apache/flink/pull/6777#discussion_r223029260
##########
File path:
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java
##########
@@ -1286,10 +1289,38 @@ private void transferAllDataFromStateHandles(
Map<StateHandleID, StreamStateHandle> stateHandleMap,
Path restoreInstancePath) throws IOException {
+ ExecutorService executorService =
Executors.newFixedThreadPool(5);
Review comment:
Number of threads should be configurable. For example, the same way as
incremental checkpointing. At least in RocksDBStateBackend constructor and
CheckpointingOptions. The default behaviour (e.g. number of threads 1) can be
as it is now, basically running in current thread.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
