Re: Implementing top() using treeReduce()

2015-06-17 Thread DB Tsai
all of them. Sincerely, DB Tsai -- Blog: https://www.dbtsai.com PGP Key ID: 0xAF08DF8D On Wed, Jun 17, 2015 at 5:15 PM, Raghav Shankar wrote: > So, I would add the assembly jar to the just the master or would I have to > add it to all th

Re: Implementing top() using treeReduce()

2015-06-17 Thread Raghav Shankar
So, I would add the assembly jar to the just the master or would I have to add it to all the slaves/workers too? Thanks, Raghav > On Jun 17, 2015, at 5:13 PM, DB Tsai wrote: > > You need to build the spark assembly with your modification and deploy > into cluster. > > Sincerely, > > DB Tsai

Re: Implementing top() using treeReduce()

2015-06-17 Thread DB Tsai
You need to build the spark assembly with your modification and deploy into cluster. Sincerely, DB Tsai -- Blog: https://www.dbtsai.com PGP Key ID: 0xAF08DF8D On Wed, Jun 17, 2015 at 5:11 PM, Raghav Shankar wrote: > I’ve implemented this

Re: Implementing top() using treeReduce()

2015-06-17 Thread Raghav Shankar
I’ve implemented this in the suggested manner. When I build Spark and attach the new spark-core jar to my eclipse project, I am able to use the new method. In order to conduct the experiments I need to launch my app on a cluster. I am using EC2. When I setup my master and slaves using the EC2 se

Re: Implementing top() using treeReduce()

2015-06-09 Thread DB Tsai
Having the following code in RDD.scala works for me. PS, in the following code, I merge the smaller queue into larger one. I wonder if this will help performance. Let me know when you do the benchmark. def treeTakeOrdered(num: Int)(implicit ord: Ordering[T]): Array[T] = withScope { if (num == 0)