i missing? What is a correct way to load a big matrix file (.mat
)from local filesystem into rdd and create a blockmatrix and do repeated
multiplication?
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/BlockMatrix-Multiplication-fails-with-Out-of-Memory-tp
mit a JIRA Issue related to the problem of block matrix
> shuffling given the blocks co-location?
>
>
>
> Best regards, Alexander
>
>
>
> *From:* Burak Yavuz [mailto:brk...@gmail.com]
> *Sent:* Wednesday, July 15, 2015 3:29 PM
>
> *To:* Ulanov, Alexander
> *Cc:* Rakesh Ch
, Alexander
Cc: Rakesh Chalasani; dev@spark.apache.org
Subject: Re: BlockMatrix multiplication
Hi Alexander,
I just noticed the error in my logic. There will always be a shuffle due to the
`cogroup`. `join` also uses cogroup, therefore a shuffle is inevitable.
However, the reduceByKey will not cause a
>
> bm.validate()
>
> val t = System.nanoTime()
>
> // multiply matrix with itself
>
> val aa = bm.multiply(bm)
>
> aa.validate()
>
> println(rows + "x" + columns + ", block:" + blockSize + "\t" +
> (System.nanoTime() - t) / 1e9)
>
>
/ 1e9)
Best regards, Alexander
From: Ulanov, Alexander
Sent: Tuesday, July 14, 2015 6:24 PM
To: 'Burak Yavuz'
Cc: Rakesh Chalasani; dev@spark.apache.org
Subject: RE: BlockMatrix multiplication
Hi Burak,
Thank you for explanation! I will try to make a diagonal block matrix and
report y
From: Burak Yavuz [mailto:brk...@gmail.com]
Sent: Tuesday, July 14, 2015 10:14 AM
To: Ulanov, Alexander
Cc: Rakesh Chalasani; dev@spark.apache.org
Subject: Re: BlockMatrix multiplication
Hi Alexander,
From your example code, using the GridPartitioner, you will have 1 column, and
5 rows. When you
sing something or using it wrong.
>
>
>
> Best regards, Alexander
>
>
>
> *From:* Rakesh Chalasani [mailto:vnit.rak...@gmail.com]
> *Sent:* Tuesday, July 14, 2015 9:05 AM
> *To:* Ulanov, Alexander
> *Cc:* dev@spark.apache.org
> *Subject:* Re: BlockMatrix multiplicatio
am missing something or
using it wrong.
Best regards, Alexander
From: Rakesh Chalasani [mailto:vnit.rak...@gmail.com]
Sent: Tuesday, July 14, 2015 9:05 AM
To: Ulanov, Alexander
Cc: dev@spark.apache.org
Subject: Re: BlockMatrix multiplication
Hi Alexander:
Aw, I missed the 'cogrou
lexander <
> alexander.ula...@hp.com> wrote:
>
>> Dear Spark developers,
>>
>>
>>
>> I am trying to perform BlockMatrix multiplication in Spark. My test is as
>> follows: 1)create a matrix of N blocks, so that each row of block matrix
>> contains onl
s a local reduce before
aggregating across nodes.
Rakesh
On Mon, Jul 13, 2015 at 9:24 PM Ulanov, Alexander
mailto:alexander.ula...@hp.com>> wrote:
Dear Spark developers,
I am trying to perform BlockMatrix multiplication in Spark. My test is as
follows: 1)create a matrix of N blocks, so
ple blocks within each partition so that reduceByKey does a local
reduce before aggregating across nodes.
Rakesh
On Mon, Jul 13, 2015 at 9:24 PM Ulanov, Alexander
wrote:
> Dear Spark developers,
>
>
>
> I am trying to perform BlockMatrix multiplication in Spark. My test is as
>
Dear Spark developers,
I am trying to perform BlockMatrix multiplication in Spark. My test is as
follows: 1)create a matrix of N blocks, so that each row of block matrix
contains only 1 block and each block resides in separate partition on separate
node, 2)transpose the block matrix and 3
12 matches
Mail list logo
