Well, I was thinking you could have avoided overwhelming your internal services by using something like Flink's async i/o operator, tuned to limit the total number of concurrent requests. That way the pipeline could have uniform parallelism without overwhelming those services, and then you'd rely on backpressure to throttle the sources. I'm not saying that would be better -- it's arguably worse to have constant backpressure.
But this point I don't understand: > running all operators at such a high scale would result in wastage of resources, even with operator chaining in place. Don't you have the same number of slots, each with the same resources, either way? Plus, you have to do more ser/de, and more networking? On Wed, May 5, 2021 at 6:08 PM vishalovercome <vis...@moengage.com> wrote: > Yes. While back-pressure would eventually ensure high throughput, hand > tuning > parallelism became necessary because the job with high source parallelism > would immediately bring down our internal services - not giving enough time > to flink to adjust the in-rate. Plus running all operators at such a high > scale would result in wastage of resources, even with operator chaining in > place. > > That's why I think more toggles are needed to make current auto-scaling > truly shine. > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ >
