Re: Spark job resource allocation best practices

Let's say that I run Spark on Mesos in fine-grained mode, and I have 12 cores and 64GB memory. I run application A on Spark, and some time after that (but before A finished) application B. How many CPUs will each of them get? *Romi Kuntsman*, *Big Data Engineer* http://www.totango.com On Tue, No

Re: Spark job resource allocation best practices

You need to install mesos on your cluster. Then you will run your spark applications by specifying mesos master (mesos://) instead of (spark://). Spark can run over Mesos in two modes: “*fine-grained*” (default) and “ *coarse-grained*”. In “*fine-grained*” mode (default), each Spark task runs as

Re: Spark job resource allocation best practices

I have a single Spark cluster, not multiple frameworks and not multiple versions. Is it relevant for my use-case? Where can I find information about exactly how to make Mesos tell Spark how many resources of the cluster to use? (instead of the default take-all) *Romi Kuntsman*, *Big Data Engineer*

Re: Spark job resource allocation best practices

You can look at different modes over here http://docs.sigmoidanalytics.com/index.php/Spark_On_Mesos#Mesos_Run_Modes These people has very good tutorial to get you started http://mesosphere.com/docs/tutorials/run-spark-on-mesos/#overview Thanks Best Regards On Tue, Nov 4, 2014 at 1:44 PM, Romi Ku

Re: Spark job resource allocation best practices

How can I configure Mesos allocation policy to share resources between all current Spark applications? I can't seem to find it in the architecture docs. *Romi Kuntsman*, *Big Data Engineer* http://www.totango.com On Tue, Nov 4, 2014 at 9:11 AM, Akhil Das wrote: > Yes. i believe Mesos is the ri

Re: Spark job resource allocation best practices

Yes. i believe Mesos is the right choice for you. http://mesos.apache.org/documentation/latest/mesos-architecture/ Thanks Best Regards On Mon, Nov 3, 2014 at 9:33 PM, Romi Kuntsman wrote: > So, as said there, static partitioning is used in "Spark’s standalone and > YARN modes, as well as the co

Re: Spark job resource allocation best practices

So, as said there, static partitioning is used in "Spark’s standalone and YARN modes, as well as the coarse-grained Mesos mode". That leaves us only with Mesos, where there is *dynamic sharing* of CPU cores. It says "when the application is not running tasks on a machine, other applications may ru

Re: Spark job resource allocation best practices

Have a look at scheduling pools . If you want more sophisticated resource allocation, then you are better of to use cluster managers like mesos or yarn Thanks Best Regards On Mon, Nov 3, 2014 at 9:10 PM, Romi Kuntsman wrote: > Hello, > >