[
https://issues.apache.org/jira/browse/FLINK-23381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-23381:
-----------------------------------
Labels: pull-request-available (was: )
> Provide backpressure (currently job fails if a limit is hit)
> ------------------------------------------------------------
>
> Key: FLINK-23381
> URL: https://issues.apache.org/jira/browse/FLINK-23381
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
> Reporter: Roman Khachatryan
> Assignee: Roman Khachatryan
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> With the current approach, job will fail if dstl.dfs.upload.max-in-flight
> (bytes) is reached.
>
> Unsetting the limit roughly matches the current behaviour for other backends:
> async phase doesn't backpressure
> (state.backend.rocksdb.checkpoint.transfer.thread.num only sets the upload
> thread pool size which uses an unbounded queue).
>
> Note that blocking caller in DfsWriter.persistInternal() will also block
> regular stream processing (because of pre-emptive writes). This may or may
> not be desired behaviour.
>
> Blocking sync phase of a snapshot can also have some issues (e.g. not being
> able to abort the checkpoint) which should be considered.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
