Hi Team, I am currently running Flink 1.16.3 with Apache Beam to run our batch jobs. We are trying to process the data with an input size of over 20 million, but unfortunately, it is failing. we want Flink to handle the parallelism based on input load automatically but even with -1 configuration, we have observed that the parallelism was set to 1 and SDK workers were not automatically scaling. we tried all sorts of configurations and their combinations but could not get the desired result.
Is there some bug concerning parallelism in the 1.16.3 version? Thanks, Suraj
