Re: Using dynamic allocation and shuffle service in Standalone Mode

2016-03-08 Thread jleaniz
You've got to start the shuffle service on all your workers. There's a script for that in the 'sbin' directory. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Using-dynamic-allocation-and-shuffle-service-in-Standalone-Mode-tp26430p26434.html Sent from the A

Re: Using dynamic allocation and shuffle service in Standalone Mode

2016-03-08 Thread Yuval Itzchakov
zchakov >> *Sent: *Tuesday, March 8, 2016 2:17 PM >> *To: *Silvio Fiorito ; >> user@spark.apache.org >> *Subject: *Re: Using dynamic allocation and shuffle service in >> Standalone Mode >> >> >> Actually, I assumed that setting the flag in the spark job would

Re: Using dynamic allocation and shuffle service in Standalone Mode

2016-03-08 Thread Andrew Or
der sbin, start-shuffle-service.sh. Run > that on each of your worker nodes. > > > > > > > > *From: *Yuval Itzchakov > *Sent: *Tuesday, March 8, 2016 2:17 PM > *To: *Silvio Fiorito ; > user@spark.apache.org > *Subject: *Re: Using dynamic allocatio

RE: Using dynamic allocation and shuffle service in Standalone Mode

2016-03-08 Thread Silvio Fiorito
<mailto:user@spark.apache.org> Subject: Re: Using dynamic allocation and shuffle service in Standalone Mode Actually, I assumed that setting the flag in the spark job would turn on the shuffle service in the workers. I now understand that assumption was wrong. Is there any way to set the flag via t

Re: Using dynamic allocation and shuffle service in Standalone Mode

2016-03-08 Thread Yuval Itzchakov
Actually, I assumed that setting the flag in the spark job would turn on the shuffle service in the workers. I now understand that assumption was wrong. Is there any way to set the flag via the driver? Or must I manually set it via spark-env.sh on each worker? On Tue, Mar 8, 2016, 20:14 Silvio F

Re: Using dynamic allocation and shuffle service in Standalone Mode

2016-03-08 Thread Suniti Singh
Please check the document for the configuration - http://spark.apache.org/docs/latest/job-scheduling.html#configuration-and-setup On Tue, Mar 8, 2016 at 10:14 AM, Silvio Fiorito < silvio.fior...@granturing.com> wrote: > You’ve started the external shuffle service on all worker nodes, correct? >

RE: Using dynamic allocation and shuffle service in Standalone Mode

2016-03-08 Thread Silvio Fiorito
You’ve started the external shuffle service on all worker nodes, correct? Can you confirm they’re still running and haven’t exited? From: Yuval.Itzchakov Sent: Tuesday, March 8, 2016 12:41 PM To: user@spark.apache.org Subject: Using dy