Arvid, thank you. So there is single instance of FIFO per async IO operator regardless of parallelism of the async IO operator? Thanks, Alexey
________________________________ From: Arvid Heise <ar...@ververica.com> Sent: Saturday, February 22, 2020 1:23:01 PM To: Alexey Trenikhun <yen...@msn.com> Cc: user@flink.apache.org <user@flink.apache.org> Subject: Re: async io parallelism Hi Alexey, the short answer is: order is preserved in all cases. Basically, ordered asyncIO maintains an internal FIFO queue where all pending elements reside. All async results are saved into this queue, but elements will only be outputted when the head element has a result. So assume you have three input records i1, i2, i3 and get the outputs asynchronously in the order o2, o1, o3 after 100 ms each, then there is no output after receiving o2, then o1 and o2 are outputted after 200 ms, and then o3 after 300 ms. Best, Arvid On Sat, Feb 22, 2020 at 2:22 AM Alexey Trenikhun <yen...@msn.com<mailto:yen...@msn.com>> wrote: Hello, Let's say, my elements are simple key-value pairs, elements are coming from Kafka, where they were partitioned by "key", then I do processing using KeyedProcessFunction (keyed by same "key"), then I enrich elements using ordered RichAsyncFunction, then output to another KeyedProcessFunction (keyed by same "key") and then write to Kafka topic, again partitioned by same "key", something like this: FlinkKafkaConsumer -> keyBy("key") -> Intake(KeyedProcessFunction) -> AsyncDataStream.orderedWait() -> keyBy("key")->Output(KeyedProcessFunction)->FlinkKafkaProducer Will it preserve order of events with same "key"? * Will Output function receive elements with same "key" in same order as they were originally in Kafka? * Will FlinkKafkaProducer writes elements with same "key" in same order as they were originally in Kafka? * Does it depend on parallelism of async IO? Documentation says "the stream order is preserved", but if there are multiple parallel instances of async function, does it mean order relative to each single instance? Or total stream order? Thanks, Alexey
