We've talked about that, but it hasn't become a priority because we haven't had a driving use case. If anyone has a good argument for "variable" resource allocation like this, please let me know.
On Sat, Jan 28, 2017 at 9:17 AM, Shuai Lin <linshuai2...@gmail.com> wrote: > An alternative behavior is to launch the job with the best resource offer >> Mesos is able to give > > > Michael has just made an excellent explanation about dynamic allocation > support in mesos. But IIUC, what you want to achieve is something like > (using RAM as an example) : "Launch each executor with at least 1GB RAM, > but if mesos offers 2GB at some moment, then launch an executor with 2GB > RAM". > > I wonder what's benefit of that? To reduce the "resource fragmentation"? > > Anyway, that is not supported at this moment. In all the supported cluster > managers of spark (mesos, yarn, standalone, and the up-to-coming spark on > kubernetes), you have to specify the cores and memory of each executor. > > It may not be supported in the future, because only mesos has the concepts > of offers because of its two-level scheduling model. > > > On Sat, Jan 28, 2017 at 1:35 AM, Ji Yan <ji...@drive.ai> wrote: > >> Dear Spark Users, >> >> Currently is there a way to dynamically allocate resources to Spark on >> Mesos? Within Spark we can specify the CPU cores, memory before running >> job. The way I understand is that the Spark job will not run if the CPU/Mem >> requirement is not met. This may lead to decrease in overall utilization of >> the cluster. An alternative behavior is to launch the job with the best >> resource offer Mesos is able to give. Is this possible with the current >> implementation? >> >> Thanks >> Ji >> >> The information in this email is confidential and may be legally >> privileged. It is intended solely for the addressee. Access to this email >> by anyone else is unauthorized. If you are not the intended recipient, any >> disclosure, copying, distribution or any action taken or omitted to be >> taken in reliance on it, is prohibited and may be unlawful. >> > > -- Michael Gummelt Software Engineer Mesosphere
