Re: Asynchronous I/O poor performance

Hi Mark, I already explained that this latency is only occurring because of the shuffle step before async IO (e.g. data is sent over network). If you replace val x : DataStream[String] = someIntegers.map( _ => s"${System.currentTimeMillis()}") with val x : DataStream[String] = someIntegers.shu

Re: Asynchronous I/O poor performance

Hi Arvid, The http client is not my buttoleneck as I said before I check the async and I have a delay until it enters to asyncinvoke about 80 ms if some can explained me why we have such big delay I have attached a sample code in my previous email can some one explain the delay Thanks On Mon, 6

Re: Asynchronous I/O poor performance

Hi Mark, Async wait operators cannot be chained to sources so the messages go through the network stack. Thus, having some latency is normal and cannot be avoided. It can be tuned though, but I don't think that this is the issue at hand as it should mostly impact latency and affect throughput less

Re: Asynchronous I/O poor performance

Hi Benchao, i have run this in the code: println(env.getConfig.getAutoWatermarkInterval) and got 200 i do fully understand how watermarks and AsyncOperator operator works, but i have decided to make a simple test that should evaluate the time it takes to enter to the asyncInvoke method and it l

Re: Asynchronous I/O poor performance

Hi Mark, could you please check if you can tune akka? Usually in async I/O, the used library uses a thread pool that becomes the actual bottleneck. If you configure async I/O to use a capacity of 100 and parallelism of 8 on one node, you also need to have ~800 threads in akka (500 might be enough

Re: Asynchronous I/O poor performance

Hi Mark, According to your data, I think the config of AsyncOperator is OK. There is one more config that might affect the throughput of AsyncOperator, it's watermark. Because unordered async operator still keeps the order between watermarks, did you use event time in your job, and if yes, what's

Re: Asynchronous I/O poor performance

Hi Benchao The capacity is 100 Parallelism is 8 Rpc req is 20ms Thanks On Sun, 5 Jul 2020, 6:16 Benchao Li, wrote: > Hi Mark, > > Could you give more details about your Flink job? > - the capacity of AsyncDataStream > - the parallelism of AsyncDataStream operator > - the time of per blocked r

Re: Asynchronous I/O poor performance

Hi Mark, Could you give more details about your Flink job? - the capacity of AsyncDataStream - the parallelism of AsyncDataStream operator - the time of per blocked rpc request Mark Zitnik 于2020年7月5日周日 上午3:48写道： > Hi > > In my flink application I need to enrich data using > AsyncDataStream.uno

Fwd: Asynchronous I/O poor performance

Hi In my flink application I need to enrich data using AsyncDataStream.unorderedWait but I am getting poor perforce at the beginning I was just working with http call, but I have switched to grpc, I running on 8 core node and getting total of 3200 events per second my service that I am using is no

Asynchronous I/O

Hi All, I have a flow which contains few external REST calls, in few different operators. As I read in documentation, using AsyncFunction is the recommended way to execute these calls. Few questions regarding that: 1. Can someone explain better the 'capacity' parameter? How it affects latency/