Hi David,
Thank you for the response. We are facing a situation of cold start for our
application. In the cold start phase, it requires a lot of parallelism to
make the busiest operator not overwhelmed so that there will be no
backpresure and no checkpoint works as normal. The problem is that such
No, there is no mechanism available for individually tuning the parallelism
of the generated operators in a SQL job. Moreover, such fine-tuning is
often counter-productive. In most cases you are better off simply setting
the overall parallelism to whatever is needed by the busiest operator(s).
Unne
Hi team,
Currently the SQL generated operator has all the same parallelism by
default, and we faced a issue that the in the case of multiple join, the
operator at later stage faces larger computation so that the overall
pipeline is back-presured and it causes checkpoint
fail(expired) occasionaly.
