Igal Shilman created FLINK-16244:
------------------------------------
Summary: Add Asynchronous operations to state lazily.
Key: FLINK-16244
URL: https://issues.apache.org/jira/browse/FLINK-16244
Project: Flink
Issue Type: Task
Components: Stateful Functions
Reporter: Igal Shilman
Currently AsyncSink would add eagerly a registered async operation.
An alternative approach would be to keep the async operations in an in memory
map and only
write them to the underlying map on snapshotState().
The rational behind this approach is the assumption that most async operations
complete between two consecutive checkpoints, and therefore adding and removing
them from the underlying state backend (rocksdb by default) is wasteful.
An implementation outline suggestion:
1. Add a LazyAsyncOperations class that keeps both an in memory map
and a MapStateHandle
this map would support add and remove and also flush() that would be called
from
snapshotState() method.
2. Use that class in AsyncSink and in AsyncMessageDecorator
3. call flush() from FunctionGroupOperator#snapshotState()
note that a special care should be taken in flash() as the current key needs to
be set
in the keyedStateBackend.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
