Re: Job initialization performance of Spark standalone mode vs YARN

;> >> On Sun, Mar 2, 2014 at 9:41 PM, polkosity wrote: >>> Thanks for the advice Mayur. >>> >>> I thought I'd report back on the performance difference... Spark standalone >>> mode has executors processing at capacity in under a second :) >>&

Re: Job initialization performance of Spark standalone mode vs YARN

k on the performance difference...  Spark standalone mode has executors processing at capacity in under a second :) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com

Re: Job initialization performance of Spark standalone mode vs YARN

to run each job). > > Our low-latency job now reads all its data from a memory cached RDD, > instead > of from HDFS seq file (upstream jobs cache resultant RDDs for downstream > jobs to read). > > > > -- > View this message in context: > http://apache-spark-user-lis

Re: Job initialization performance of Spark standalone mode vs YARN

tream jobs to read). -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Job-initialization-performance-of-Spark-standalone-mode-vs-YARN-tp2016p2384.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Job initialization performance of Spark standalone mode vs YARN

in memory, performance > is 4 times better. We didn't expect that much of an improvement! > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Job-initialization-performance-of-Spark-standalone-mode-vs-YARN-tp2016p2340.html >

Re: Job initialization performance of Spark standalone mode vs YARN

After changing to reuse spark context and cache RDDs in memory, performance is 4 times better. We didn't expect that much of an improvement! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Job-initialization-performance-of-Spark-standalone-mode-vs

Re: Job initialization performance of Spark standalone mode vs YARN

esystem, and could offer their views on using it? >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Job-initialization-performance-of-Spark-standalone-mode-vs-YARN-tp2016p2265.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> > >

Re: Job initialization performance of Spark standalone mode vs YARN

as anyone had any experience running Spark on Tachyon > in-memory filesystem, and could offer their views on using it? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Job-initialization-performance-of-Spark-standalone-mode-vs-YAR

Re: Job initialization performance of Spark standalone mode vs YARN

erience running Spark on Tachyon > in-memory filesystem, and could offer their views on using it? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Job-initialization-performance-of-Spark-standalone-mode-vs-YARN-tp2016p2265.html &g

Re: Job initialization performance of Spark standalone mode vs YARN

-initialization-performance-of-Spark-standalone-mode-vs-YARN-tp2016p2265.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Job initialization performance of Spark standalone mode vs YARN

> http://apache-spark-user-list.1001560.n3.nabble.com/Job-initialization-performance-of-Spark-standalone-mode-vs-YARN-tp2016p2263.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >

Re: Job initialization performance of Spark standalone mode vs YARN

ce-of-Spark-standalone-mode-vs-YARN-tp2016p2263.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Job initialization performance of Spark standalone mode vs YARN

dalone > mode has executors processing at capacity in under a second :) > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Job-initialization-performance-of-Spark-standalone-mode-vs-YARN-tp2016p2243.html > Sent from the Apach

Re: Job initialization performance of Spark standalone mode vs YARN

ecutors processing at capacity in under a second :) > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Job-initialization-performance-of-Spark-standalone-mode-vs-YARN-tp2016p2243.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >

Re: Job initialization performance of Spark standalone mode vs YARN

Thanks for the advice Mayur. I thought I'd report back on the performance difference... Spark standalone mode has executors processing at capacity in under a second :) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Job-initialization-performance-of-

Re: Job initialization performance of Spark standalone mode vs YARN

? > Yarn has much more granular control over the cluster resources. You can also look into Mesos for management which will be much faster than Yarn for now. > > Thanks. > - Dan > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com

Job initialization performance of Spark standalone mode vs YARN

- Dan -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Job-initialization-performance-of-Spark-standalone-mode-vs-YARN-tp2016.html Sent from the Apache Spark User List mailing list archive at Nabble.com.