Hi Burak,
That’s interesting. I’ll try and give it a go.
Eilidh
On 14 Nov 2015, at 04:19, Burak Yavuz wrote:
> Hi,
>
> The BlockMatrix multiplication should be much more efficient on the current
> master (and will be available with Spark 1.6). Could you please give that a
> try if you have
Hi,
The BlockMatrix multiplication should be much more efficient on the current
master (and will be available with Spark 1.6). Could you please give that a
try if you have the chance?
Thanks,
Burak
On Fri, Nov 13, 2015 at 10:11 AM, Sabarish Sasidharan <
sabarish.sasidha...@manthan.com> wrote:
>
Hi Eilidh
Because you are multiplying with the transpose you don't have to
necessarily build the right side of the matrix. I hope you see that. You
can broadcast blocks of the indexed row matrix to itself and achieve the
multiplication.
But for similarity computation you might want to use some a
Hi Sab,
Thanks for your response. We’re thinking of trying a bigger cluster, because we
just started with 2 nodes. What we really want to know is whether the code will
scale up with larger matrices and more nodes. I’d be interested to hear how
large a matrix multiplication you managed to do?
I
We have done this by blocking but without using BlockMatrix. We used our
own blocking mechanism because BlockMatrix didn't exist in Spark 1.2. What
is the size of your block? How much memory are you giving to the executors?
I assume you are running on YARN, if so you would want to make sure your
ya
