Mailed our list - will send it to Spark Dev
On Fri, Sep 5, 2014 at 11:28 AM, Rajat Gupta <rgu...@qubole.com> wrote: > +1 on this. First step to more automated autoscaling of spark application > master... > > > On Fri, Sep 5, 2014 at 12:56 AM, Praveen Seluka <psel...@qubole.com> > wrote: > >> +user >> >> >> >> On Thu, Sep 4, 2014 at 10:53 PM, Praveen Seluka <psel...@qubole.com> >> wrote: >> >>> Spark on Yarn has static allocation of resources. >>> https://issues.apache.org/jira/browse/SPARK-3174 - This JIRA by Sandy >>> is about adding and removing executors dynamically based on load. Even >>> before doing this, can we expose and interface to add/remove executors. >>> This will be very useful in Spark Shell. >>> >>> User can launch Spark Shell with just few executors. And later user >>> wants more executors to be added, user could call some API like >>> sc.addExecutor(count). We could similarly have removeExecutor API too. I >>> have been thinking to work on this. Later, SPARK-3174 JIRA can add >>> intelligence on when to add/remove depending on load. Right now, user will >>> get control over adding and removing executor at runtime. >>> >>> Thoughts? >>> >>> Praveen >>> >> >> >
