I was wondering if you could actually really use AsyncWaitOperator in the following way.
- Use a rather big timeout (so if callbacks usually take 1s, use 10). - Use UNORDERED mode. - Use a rather big queue size that would not cause any backpressure (you could just experiment with different settings). Then, you'd probably get to the operator that you would need to implement manually anyways. - Requests come in a specific order, that order is retained when calling the external library. - Results are immediately returned (depending on your watermark settings) resulting in no additional latency (because of UNORDERED). - The big timeouts guarantee that you will not dismiss a certain input too quickly, if the callback takes longer than usual. It will clean up all elements from state that have no callbacks after the given time though. - The big queue size will avoid backpressure resulting from many pending requests without response. Let's say you have 100 requests per second and a timeout of 10s, that means a queue size of 1000 would allow all incoming requests to be processed almost instantly (ignored the actual callbacks that decrease the needed queue size as you said it to be a rather rare event) On Thu, Apr 9, 2020 at 11:09 AM Salva Alcántara <salcantara...@gmail.com> wrote: > Perfectly understood, thanks a lot for your reply/patience . I will take a > look at AsyncWaitOperator and adapt from there if I really need that. > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ > -- Arvid Heise | Senior Java Developer <https://www.ververica.com/> Follow us @VervericaData -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng
