You should also check the available YARN resources, overall the number of
containers can be allocated is restricted by Yarn resources. I guess here
your Yarn cluster resources can only allocate 3 containers, even if you set
the initial number to 10, still it cannot be satisfied.
On Wed, Jan 27, 2
Hi,
As per spark documentation for spark's Dynamic Resource Allocation. I did
the following to enable shuffle/ Dynamic allocation service:
A) Added the following lines to "spark-defaults.conf"
enabling dynamic resource allocation and shuffle service
spark.dynamicAllocation.enabled=true
spark.dyn
