Hi Chawla, One possible reason is that Mesos fine grain mode also takes up cores to run the executor per host, so if you have 20 agents running Fine grained executor it will take up 20 cores while it's still running.
Tim On Fri, Dec 16, 2016 at 8:41 AM, Chawla,Sumit <sumitkcha...@gmail.com> wrote: > Hi > > I am using Spark 1.6. I have one query about Fine Grained model in Spark. > I have a simple Spark application which transforms A -> B. Its a single > stage application. To begin the program, It starts with 48 partitions. > When the program starts running, in mesos UI it shows 48 tasks and 48 CPUs > allocated to job. Now as the tasks get done, the number of active tasks > number starts decreasing. How ever, the number of CPUs does not decrease > propotionally. When the job was about to finish, there was a single > remaininig task, however CPU count was still 20. > > My questions, is why there is no one to one mapping between tasks and cpus > in Fine grained? How can these CPUs be released when the job is done, so > that other jobs can start. > > > Regards > Sumit Chawla --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
