Hi So we are running a beam pipeline that uses flink as its execution engine. We are currently on flink1.8 So per the flink documentation I see that there is an option that allows u to set
Parallelism and maxParallelism. We actually want to set both so that we can dynamically scale the pipeline if there is back pressure on it. I want to clarify a few doubts I had – 1. Does the maxParallelism only work for stateful operators or does it work for all the operators? 1. Also is the setting either or ? Like we can only set parallelism or maxParallelism? Or is it that we can set both? 1. Because, Currently I have the parallelism at 8 and maxparallelism at 32 but the UI only shows me that it is using parallelism of 8. It would be great if someone helps me understand the exact behavior of these parameters. Thanks Akshay Iyangar
