Hello Everyone, I am using a series of 6 Async Operators in my application. Each operator is using AsyncHttpClient to make some external API call. Each Async Operator makes only one call to external API for a message. Capacity is set t 1000 for each parallelism. Approximately, we are getting response in 1000ms with correction of 20ms. We see that, at parallelism of 8 for async, only 5-8 messages are inside single parallelism of an async operator. I am trying to figure out what decides how many messages will go inside the Async Operator.
Topology: Source - Process - Async(parallelism 8) - Async(parallelism 8) - Async(parallelism 8) - Async(parallelism 8) - Async(parallelism 8) - Async(parallelism 8) - process - sink. Resources: 1vcore, 4GB ram, running on AWS Kinesis Analytics. Metrics: CPU utilization 22%, Memory utilization: 50-58%. It will be helpful if I can get some articles on how Flink decides on how many messages will enter into an Async Operator when the capacity is specified as 1000 but available memory is only 4GB. Thanks, Sanket Agrawal
