Re: Driver staggering task launch times

Ah, nevermind, I don't know anything about scheduling tasks in YARN. On Thu, Aug 13, 2015 at 11:03 PM, Ara Vartanian wrote: > I’m running on Yarn. > > On Aug 13, 2015, at 10:58 PM, Philip Weaver > wrote: > > Are you running on mesos, yarn or standalone? If you're on mesos, are you > using coars

Re: Driver staggering task launch times

I’m running on Yarn. > On Aug 13, 2015, at 10:58 PM, Philip Weaver wrote: > > Are you running on mesos, yarn or standalone? If you're on mesos, are you > using coarse grain or fine grained mode? > > On Thu, Aug 13, 2015 at 10:13 PM, Ara Vartanian > wrote: > I’m obs

Re: Driver staggering task launch times

Are you running on mesos, yarn or standalone? If you're on mesos, are you using coarse grain or fine grained mode? On Thu, Aug 13, 2015 at 10:13 PM, Ara Vartanian wrote: > I’m observing an unusual situation where my step duration increases as I > add further executors to my cluster. My algorithm

Driver staggering task launch times

I’m observing an unusual situation where my step duration increases as I add further executors to my cluster. My algorithm is fully data parallelizable into a map phase, followed by a reduce step at the end that amounts to matrix addition. So I’ve kicked a cluster of, say, 100 executors with 4 c