The drive has a heuristic mechanism to decide the number of executors in the run-time according the pending tasks. You could enable with configuration, you could refer to spark document to find the details.
2015-05-27 15:00 GMT+08:00 canan chen <[email protected]>: > How does the dynamic allocation works ? I mean does it related > with parallelism of my RDD and how does driver know how many executor it > needs ? > > On Wed, May 27, 2015 at 2:49 PM, Saisai Shao <[email protected]> > wrote: > >> It depends on how you use Spark, if you use Spark with Yarn and enable >> dynamic allocation, the number of executor is not fixed, will change >> dynamically according to the load. >> >> Thanks >> Jerry >> >> 2015-05-27 14:44 GMT+08:00 canan chen <[email protected]>: >> >>> It seems the executor number is fixed for the standalone mode, not sure >>> other modes. >>> >> >> >
