Re: Handling tree reduction algorithm with Spark in parallel

Thanks Matei, will check out the MLLib implementation. On Wed, Oct 1, 2014 at 2:24 PM, Andy Twigg wrote: > Yes, that makes sense. It's similar to the all reduce pattern in vw. > > > On Wednesday, 1 October 2014, Matei Zaharia > wrote: > >> Some of the MLlib algorithms do tree reduction in 1.1:

Re: Handling tree reduction algorithm with Spark in parallel

Yes, that makes sense. It's similar to the all reduce pattern in vw. On Wednesday, 1 October 2014, Matei Zaharia wrote: > Some of the MLlib algorithms do tree reduction in 1.1: > http://databricks.com/blog/2014/09/22/spark-1-1-mllib-performance-improvements.html. > You can check out how they imp

Re: Handling tree reduction algorithm with Spark in parallel

Some of the MLlib algorithms do tree reduction in 1.1: http://databricks.com/blog/2014/09/22/spark-1-1-mllib-performance-improvements.html. You can check out how they implemented it -- it is a series of reduce operations. Matei On Oct 1, 2014, at 11:02 AM, Boromir Widas wrote: > Thanks a lot

Re: Handling tree reduction algorithm with Spark in parallel

Thanks a lot Andy and Debashish, your suggestions were of great help. On Tue, Sep 30, 2014 at 6:44 PM, Debasish Das wrote: > If the tree is too big build it on graphxbut it will need thorough > analysis so that the partitions are well balanced... > > On Tue, Sep 30, 2014 at 2:45 PM, Andy Twi

Re: Handling tree reduction algorithm with Spark in parallel

If the tree is too big build it on graphxbut it will need thorough analysis so that the partitions are well balanced... On Tue, Sep 30, 2014 at 2:45 PM, Andy Twigg wrote: > Hi Boromir, > > Assuming the tree fits in memory, and what you want to do is parallelize > the computation, the 'obviou

Re: Handling tree reduction algorithm with Spark in parallel

Hi Boromir, Assuming the tree fits in memory, and what you want to do is parallelize the computation, the 'obvious' way is the following: * broadcast the tree T to each worker (ok since it fits in memory) * construct an RDD for the deepest level - each element in the RDD is (parent,data_at_node)

Handling tree reduction algorithm with Spark in parallel

Hello Folks, I have been trying to implement a tree reduction algorithm recently in spark but could not find suitable parallel operations. Assuming I have a general tree like the following - I have to do the following - 1) Do some computation at each leaf node to get an array of doubles.(This c