Oh, thanks for mentioning that, it looks l dynamic allocation on Kubernetes works in client mode in Spark 3.0.0. I just had to set the following configurations:
spark.dynamicAllocation.enabled=true spark.dynamicAllocation.shuffleTracking.enabled=true to enable dynamic allocation and disable the need for the external shuffle service (which looks like it is experimental right now). My executor pods couldn't connect to the external shuffle service when it was enabled. This seems to be working okay for me. Thanks, Steven On Tue, May 12, 2020 at 4:42 AM Pradeepta Choudhury < pradeeptachoudhu...@gmail.com> wrote: > Hey guys i was able to run dynamic scaling in both cluster and client mode > . would document and send it over this weekend > > On Tue 12 May, 2020, 1:26 PM Roland Johann, <roland.joh...@phenetic.io> > wrote: > >> Hi all, >> >> don’t want to interrupt the conversation but are keen where I can find >> information regarding dynamic allocation on kubernetes. As far as I know >> the docs just point to future work. >> >> Thanks a lot, >> Roland >> >> >> >> Am 12.05.2020 um 09:25 schrieb Steven Stetzler <steven.stetz...@gmail.com >> >: >> >> Hi all, >> >> I am interested in this as well. My use-case could benefit from dynamic >> executor scaling but we are restricted to using client mode since we are >> only using Spark shells. >> >> Could anyone help me understand the barriers to getting dynamic executor >> scaling to work in client mode on Kubernetes? >> >> Thanks, >> Steven >> >> On Sat, May 9, 2020 at 9:48 AM Pradeepta Choudhury < >> pradeeptachoudhu...@gmail.com> wrote: >> >>> Hiii , >>> >>> The dynamic executor scalling is working fine for spark on kubernetes >>> (latest from spark master repository ) in cluster mode . is the dynamic >>> executor scalling available for client mode ? if yes where can i find the >>> usage doc for same . >>> If no is there any PR open for this ? >>> >>> Thanks , >>> Pradeepta >>> >> >>
