Hi Vishal, WRT “bring down our internal services” - a common pattern with making requests to external services is to measure latency, and throttle (delay) requests in response to increased latency.
You’ll see this discussed frequently on web crawling forums as an auto-tuning approach. Typically there’s a steady increase in latency as load on the service increases. The trick is throttling soon enough before you hit the “elbow” where a service effectively falls over. — Ken > On May 5, 2021, at 9:08 AM, vishalovercome <vis...@moengage.com> wrote: > > Yes. While back-pressure would eventually ensure high throughput, hand tuning > parallelism became necessary because the job with high source parallelism > would immediately bring down our internal services - not giving enough time > to flink to adjust the in-rate. Plus running all operators at such a high > scale would result in wastage of resources, even with operator chaining in > place. > > That's why I think more toggles are needed to make current auto-scaling > truly shine. > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ -------------------------- Ken Krugler http://www.scaleunlimited.com Custom big data solutions Flink, Pinot, Solr, Elasticsearch
