Hi,all
I'm using Spark 2.0.0 to train a model with 1000w+ parameters, about 500GB
data. The treeAggregate is used to aggregate the gradient, when I set the
depth = 2 or 3, it works, and depth equals to 3 is faster.
So I set depth = 4 to obtain better performance, but now some executors
will be OOM
llocationManager=DEBUG" to log4j
> conf to expose more details, then maybe you could dig out some clues.
>
>
> Thanks
> Saisai
>
> On Thu, Mar 10, 2016 at 10:18 AM, Jy Chen wrote:
>
>> Sorry,the last configuration is also --conf
>> spark.dynamicAllocation.cach
Sorry,the last configuration is also --conf
spark.dynamicAllocation.cachedExecutorIdleTimeout=60s, "--conf" was lost
when I copied it to mail.
-- Forwarded message ------
From: Jy Chen
Date: 2016-03-10 10:09 GMT+08:00
Subject: Re: Dynamic allocation doesn't work on
the configurations of dynamic allocation so we
> could know better.
>
> On Wed, Mar 9, 2016 at 4:29 PM, Jy Chen wrote:
>
>> Hello everyone:
>>
>> I'm trying the dynamic allocation in Spark on YARN. I have followed
>> configuration steps and started the shuffle service.
Hello everyone:
I'm trying the dynamic allocation in Spark on YARN. I have followed
configuration steps and started the shuffle service.
Now it can request executors when the workload is heavy but it cannot
remove executors. I try to open the spark shell and don’t run any command,
no executor is
