I haven't run on mesos before, but I do run on yarn. The performance differences are going to be in how long it takes you go get the Executors allocated. On yarn that is going to depend on the cluster setup. If you have dedicated resources to a queue where you are running your spark job the overhead is pretty minimal. Now if your cluster is multi-tenant and is really busy and you allow other queues are using your capacity it could take some time. It is also possible to run into the situation where the memory of the nodemanagers get fragmented and you don't have any slots big enough for you so you have to wait for other applications to finish. Again this mostly depends on the setup, how big of containers you need for Spark, etc.
Tom On Thursday, April 10, 2014 11:12 AM, Flavio Pompermaier <pomperma...@okkam.it> wrote: Thank you for the reply Mayur, it would be nice to have a comparison about that. I hope one day it will be available, or to have the time to test it myself :) So you're using Mesos for the moment, right? Which are the main differences in you experience? YARN seems to be more flexible and interoperable with other frameworks..am I wrong? Best, Flavio On Thu, Apr 10, 2014 at 5:55 PM, Mayur Rustagi <mayur.rust...@gmail.com> wrote: I've had better luck with standalone in terms of speed & latency. I think thr is impact but not really very high. Bigger impact is towards being able to manage resources & share cluster. > > >Mayur Rustagi >Ph: +1 (760) 203 3257 >http://www.sigmoidanalytics.com >@mayur_rustagi > > > > > >On Wed, Apr 9, 2014 at 12:10 AM, Flavio Pompermaier <pomperma...@okkam.it> >wrote: > >Hi to everybody, >>I'm new to Spark and I'd like to know if running Spark on top of YARN or >>Mesos could affect (and how much) its performance. Is there any doc about >>this? >> >> >>Best, >>Flavio
