+user
On Thu, Sep 4, 2014 at 10:53 PM, Praveen Seluka <psel...@qubole.com> wrote: > Spark on Yarn has static allocation of resources. > https://issues.apache.org/jira/browse/SPARK-3174 - This JIRA by Sandy is > about adding and removing executors dynamically based on load. Even before > doing this, can we expose and interface to add/remove executors. This will > be very useful in Spark Shell. > > User can launch Spark Shell with just few executors. And later user wants > more executors to be added, user could call some API like > sc.addExecutor(count). We could similarly have removeExecutor API too. I > have been thinking to work on this. Later, SPARK-3174 JIRA can add > intelligence on when to add/remove depending on load. Right now, user will > get control over adding and removing executor at runtime. > > Thoughts? > > Praveen >
