Hi, My Spark version is 1.5.1 with Hadoop 2.5.0-cdh5.2.0. These are my configurations of dynamic allocation: --master yarn-client --conf spark.dynamicAllocation.enabled=true --conf spark.shuffle.service.enabled=true --conf spark.dynamicAllocation.minExecutors=0 --conf spark.dynamicAllocation.initialExecutors=10 --conf spark.dynamicAllocation.executorIdleTimeout=60s spark.dynamicAllocation.cachedExecutorIdleTimeout=60s
At first,it will remove 2 executors and then no more executors will be removed. Thanks 2016-03-09 17:24 GMT+08:00 Saisai Shao <sai.sai.s...@gmail.com>: > Would you please send out the configurations of dynamic allocation so we > could know better. > > On Wed, Mar 9, 2016 at 4:29 PM, Jy Chen <chen.wah...@gmail.com> wrote: > >> Hello everyone: >> >> I'm trying the dynamic allocation in Spark on YARN. I have followed >> configuration steps and started the shuffle service. >> >> Now it can request executors when the workload is heavy but it cannot >> remove executors. I try to open the spark shell and don’t run any command, >> no executor is removed after spark.dynamicAllocation.executorIdleTimeout >> interval. >> Am I missing something? >> >> Thanks. >> >> >
