Re: Can not set spark dynamic resource allocation

2016-05-27 Thread Cui, Weifeng
Newberger > >-Original Message- >From: Cui, Weifeng [mailto:weife...@a9.com] >Sent: Friday, May 20, 2016 4:26 PM >To: Marcelo Vanzin >Cc: Ted Yu; Rodrick Brown; user; Zhao, Jun; Aulakh, Sahib; Song, Yiwei >Subject: Re: Can not set spark dynamic resource allocation > >Sorry

RE: Can not set spark dynamic resource allocation

2016-05-20 Thread David Newberger
- From: Cui, Weifeng [mailto:weife...@a9.com] Sent: Friday, May 20, 2016 4:26 PM To: Marcelo Vanzin Cc: Ted Yu; Rodrick Brown; user; Zhao, Jun; Aulakh, Sahib; Song, Yiwei Subject: Re: Can not set spark dynamic resource allocation Sorry, here is the node-manager log. application_1463692924309_

Re: Can not set spark dynamic resource allocation

2016-05-20 Thread Cui, Weifeng
>> >> >> From: "Aulakh, Sahib" >> Date: Friday, May 20, 2016 at 12:43 PM >> To: Ted Yu >> Cc: Rodrick Brown , Cui Weifeng >> , user , "Zhao, Jun" >> >> Subject: Re: Can not set spark dynamic resource allocation >> &

Re: Can not set spark dynamic resource allocation

2016-05-20 Thread Marcelo Vanzin
gt; > > > Thanks, > Weifeng > > > > > > From: "Aulakh, Sahib" > Date: Friday, May 20, 2016 at 12:43 PM > To: Ted Yu > Cc: Rodrick Brown , Cui Weifeng > , user , "Zhao, Jun" > > Subject: Re: Can not set spark dynamic resource allocation >

Re: Can not set spark dynamic resource allocation

2016-05-20 Thread Cui, Weifeng
Here is the application log for this spark job. http://pastebin.com/2UJS9L4e Thanks, Weifeng From: "Aulakh, Sahib" Date: Friday, May 20, 2016 at 12:43 PM To: Ted Yu Cc: Rodrick Brown , Cui Weifeng , user , "Zhao, Jun" Subject: Re: Can not set spark dynamic resource a

Re: Can not set spark dynamic resource allocation

2016-05-20 Thread Aulakh, Sahib
Yes it is yarn. We have configured spark shuffle service w yarn node manager but something must be off. We will send u app log on paste bin. Sent from my iPhone On May 20, 2016, at 12:35 PM, Ted Yu mailto:yuzhih...@gmail.com>> wrote: Since yarn-site.xml was cited, I assume the cluster runs YA

Re: Can not set spark dynamic resource allocation

2016-05-20 Thread Ted Yu
Since yarn-site.xml was cited, I assume the cluster runs YARN. On Fri, May 20, 2016 at 12:30 PM, Rodrick Brown wrote: > Is this Yarn or Mesos? For the later you need to start an external shuffle > service. > > Get Outlook for iOS > > > > > On Fri, May 20, 2016 at 11:48 AM

Re: Can not set spark dynamic resource allocation

2016-05-20 Thread Rodrick Brown
Is this Yarn or Mesos? For the later you need to start an external shuffle service.  Get Outlook for iOS On Fri, May 20, 2016 at 11:48 AM -0700, "Cui, Weifeng" wrote: Hi guys,   Our team has a hadoop 2.6.0 cluster with Spark 1.6.1. We want to set dynamic resource a

Re: Can not set spark dynamic resource allocation

2016-05-20 Thread Ted Yu
Can you retrieve the log for application_1463681113470_0006 and pastebin it ? Thanks On Fri, May 20, 2016 at 11:48 AM, Cui, Weifeng wrote: > Hi guys, > > > > Our team has a hadoop 2.6.0 cluster with Spark 1.6.1. We want to set > dynamic resource allocation for spark and we followed the followin

Can not set spark dynamic resource allocation

2016-05-20 Thread Cui, Weifeng
Hi guys, Our team has a hadoop 2.6.0 cluster with Spark 1.6.1. We want to set dynamic resource allocation for spark and we followed the following link. After the changes, all spark jobs failed. https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation This test was