Hey Taher, To answer question 2 and 3 - per the flink documentation[1] in-flight requests are stored in checkpoints and only the unsent records will be sent when recovering from a failure.
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/asyncio/#fault-tolerance-guarantees Best, Eric On Tue, Aug 6, 2024 at 6:47 AM Taher Koitawala <taher...@gmail.com> wrote: > Hi All, > I have a use case where I read from Kafkf and have some records > that I want to submit from Flink to a rest service with throttling, I > thought that the AsyncIO operator would be the best for such use cases, > however I wanted to know if AsyncIO is a stateful operator? > > What I want to know is: > > 1. Backpressure Management: > 1. I expect that the operator will not cause a backpressure Flink, > TPM of the rest service is approx 10000, so I assume AsyncIO will > buffer > records to state > 2. In case of failures and restart will the AsyncIO operator send the > last lost set of records only? > 3. If let's say 200 records are to be sent and only 100 are sent and > flink fails on restart does it send only 100 more? > > If AsyncIO is not the right operator please can you tell me what we can use > instead to achieve all this. > > Regards, > Taher Koitawala >
