Thank Lincoln for the proposal.
## Motivation:
> asyncInvoke and callback functions are executed synchronously by the main
thread, which is not suitable adding long time blocking operations, and
introducing additional thread will bring extra complexity for users
According to the documentation of AsyncFunction:
> For each #asyncInvoke, an async io operation can be triggered, and once
it has been done, the result can be collected by calling {@link
ResultFuture#complete}. For each async operation, its context is stored in
the operator immediately after invoking #asyncInvoke, avoiding blocking for
each stream input as long as the internal buffer is not full.
The real external call should happen in the asynchronous thread.
My question is: If the user makes a retry in this asynchronous thread by
themselves, is there a difference between this and the current FLIP's?
## Block Main Thread
If I understand correctly, the retry queue will be put into ListState, this
state is OperatorState? As far as I know, OperatorState does not have the
ability to store a lot of data.
So after we need to retry more data, we should need to block the main
thread? What is the maximum size of the default retry queue?
Best,
Jingsong
On Thu, May 12, 2022 at 8:56 PM Lincoln Lee <lincoln.8...@gmail.com> wrote:
> Dear Flink developers,
>
> I would like to open a discussion on FLIP 232 [1], for an extension of
> AsyncWaitOperator to support retry for user's asyncFunction.
>
> To do so, new user interface will added to define the trigger condition for
> retry and when should retry. Internally, a delayed retry mechanism will be
> introduced.
>
> There's PoC for this FLIP [2][3], thanks Yun Gao for offline discussions
> and valuable comments.
> The new feature is backwards compatible that can recover from state which
> was generated by prior flink versions, and if no retry strategy enabled the
> behavior is as before.
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211883963
> [2] based on timer trigger
> https://github.com/lincoln-lil/flink/pull/new/async-retry-timer
> [3] based on DelayQueue with pull fashion
> https://github.com/lincoln-lil/flink/pull/new/async-op-retry
>
>
> Best,
> Lincoln Lee
>
