Sorry,the last configuration is also --conf spark.dynamicAllocation.cachedExecutorIdleTimeout=60s, "--conf" was lost when I copied it to mail.
---------- Forwarded message ---------- From: Jy Chen <chen.wah...@gmail.com> Date: 2016-03-10 10:09 GMT+08:00 Subject: Re: Dynamic allocation doesn't work on YARN To: Saisai Shao <sai.sai.s...@gmail.com>, user@spark.apache.org Hi, My Spark version is 1.5.1 with Hadoop 2.5.0-cdh5.2.0. These are my configurations of dynamic allocation: --master yarn-client --conf spark.dynamicAllocation.enabled=true --conf spark.shuffle.service.enabled=true --conf spark.dynamicAllocation.minExecutors=0 --conf spark.dynamicAllocation.initialExecutors=10 --conf spark.dynamicAllocation.executorIdleTimeout=60s spark.dynamicAllocation.cachedExecutorIdleTimeout=60s At first,it will remove 2 executors and then no more executors will be removed. Thanks 2016-03-09 17:24 GMT+08:00 Saisai Shao <sai.sai.s...@gmail.com>: > Would you please send out the configurations of dynamic allocation so we > could know better. > > On Wed, Mar 9, 2016 at 4:29 PM, Jy Chen <chen.wah...@gmail.com> wrote: > >> Hello everyone: >> >> I'm trying the dynamic allocation in Spark on YARN. I have followed >> configuration steps and started the shuffle service. >> >> Now it can request executors when the workload is heavy but it cannot >> remove executors. I try to open the spark shell and don’t run any command, >> no executor is removed after spark.dynamicAllocation.executorIdleTimeout >> interval. >> Am I missing something? >> >> Thanks. >> >> >
