Hi Nikhil, Spark on Kubernetes supports dynamic allocation using shuffle tracking feature instead of the external shuffle service. In order to enable dynamic allocation, you should set these two configs as true: spark.dynamicAllocation.enabled and spark.dynamicAllocation.shuffleTracking.enabled
Regards, Shrikant On Tue, 25 Oct 2022 at 10:44 PM, Nikhil Goyal <nownik...@gmail.com> wrote: > Hi folks, > When running spark on Kubernetes is it possible to use dynamic allocation? > Some blog posts > <https://spot.io/blog/setting-up-managing-monitoring-spark-on-kubernetes/> > mentioned that dynamic allocation is available, however I am not sure how > it works. Spark official docs > <https://spark.apache.org/docs/latest/running-on-kubernetes.html#future-work> > say that shuffle service is not yet available. > > Thanks > > Nikhil > -- Regards, Shrikant Prasad
