Re: TallSkinnyQR

2017年10月7日 上午5:29，Iman Mohtashemi 写道：Hi guys,Here is another problem I encountered using the tallskinny QR. I've attached some clear documentation of the problem. I posted it on the forum but I'm not sure if it went throughBest regards,ImanOn Fri, Dec 30, 2016 at 9:22 AM Sean Owen

Re: TallSkinnyQR

There are no changes to Spark at all here. See my workaround below. On Fri, Dec 30, 2016, 17:18 Iman Mohtashemi wrote: > Hi guys, > Are your changes/bug fixes reflected in the Spark 2.1 release? > Iman > > On Dec 2, 2016 3:03 PM, "Iman Mohtashemi" > wrote: > > Thanks again! This is very helpful

Re: TallSkinnyQR

Hi guys, Are your changes/bug fixes reflected in the Spark 2.1 release? Iman On Dec 2, 2016 3:03 PM, "Iman Mohtashemi" wrote: > Thanks again! This is very helpful! > Best regards, > Iman > > On Dec 2, 2016 2:49 PM, "Huamin Li" <3eri...@gmail.com> wrote: > >> Hi Iman, >> >> You can get my code fr

Re: TallSkinnyQR

Thanks again! This is very helpful! Best regards, Iman On Dec 2, 2016 2:49 PM, "Huamin Li" <3eri...@gmail.com> wrote: > Hi Iman, > > You can get my code from https://github.com/hl475/svd/tree/testSVD. In > additional to fix the index issue for IndexedRowMatrix ( > https://issues.apache.org/jira/b

Re: TallSkinnyQR

Hi Iman, You can get my code from https://github.com/hl475/svd/tree/testSVD. In additional to fix the index issue for IndexedRowMatrix ( https://issues.apache.org/jira/browse/SPARK-8614), I have made some the following changes as well: (1) Add tallSkinnySVD and computeSVDbyGram to indexedRowMatri

Re: TallSkinnyQR

Great thanks! Where can I get the latest with the bug fixes? best regards, Iman On Fri, Dec 2, 2016 at 10:54 AM Huamin Li <3eri...@gmail.com> wrote: > Hi, > > There seems to be a bug in the section of code that converts the RowMatrix > format back into indexedRowMatrix format. > > For RowMatrix,

Re: TallSkinnyQR

Hi, There seems to be a bug in the section of code that converts the RowMatrix format back into indexedRowMatrix format. For RowMatrix, I think the singular values and right singular vectors (not the left singular vectors U) that computeSVD computes are correct when using multiple executors/machi

Re: TallSkinnyQR

I have a different question that might be trivial for you (although not to me :)) Maybe you can answer this? Here is a MapReduce Example implemented in Java. It reads each line of text and for each word in the line of text determines if it starts with an upper case. If so, it creates a key value p

Re: TallSkinnyQR

Ok thanks. On Fri, Dec 2, 2016 at 8:19 AM Sean Owen wrote: > I tried, but enforcing the ordering changed a fair bit of behavior and I > gave up. I think the way to think of it is: a RowMatrix has whatever > ordering you made it with, so you need to give it ordered rows if you're > going to use a

Re: TallSkinnyQR

I tried, but enforcing the ordering changed a fair bit of behavior and I gave up. I think the way to think of it is: a RowMatrix has whatever ordering you made it with, so you need to give it ordered rows if you're going to use a method like the QR decomposition. That works. I don't think the QR me

Re: TallSkinnyQR

Hi guys, Was this bug ever resolved? Iman On Fri, Nov 11, 2016 at 9:59 AM Iman Mohtashemi wrote: > Yes this would be helpful, otherwise the Q part of the decomposition is > useless. One can use that to solve the system by transposing it and > multiplying with b and solving for x (Ax = b) where

Re: TallSkinnyQR

Yes this would be helpful, otherwise the Q part of the decomposition is useless. One can use that to solve the system by transposing it and multiplying with b and solving for x (Ax = b) where A = R and b = Qt*b since the Upper triangular matrix is correctly available (R) On Fri, Nov 11, 2016 at 3

Re: TallSkinnyQR

@Xiangrui / @Joseph, do you think it would be reasonable to have CoordinateMatrix sort the rows it creates to make an IndexedRowMatrix? in order to make the ultimate output of toRowMatrix less surprising when it's not ordered? On Tue, Nov 8, 2016 at 3:29 PM Sean Owen wrote: > I think the problem

Re: TallSkinnyQR

Thanks Sean! Let me take a look! Iman On Nov 8, 2016 7:29 AM, "Sean Owen" wrote: > I think the problem here is that IndexedRowMatrix.toRowMatrix does *not* > result in a RowMatrix with rows in order of their indices, necessarily: > > // Drop its row indices. > RowMatrix rowMat = indexedRowMatrix

Re: TallSkinnyQR

I think the problem here is that IndexedRowMatrix.toRowMatrix does *not* result in a RowMatrix with rows in order of their indices, necessarily: // Drop its row indices. RowMatrix rowMat = indexedRowMatrix.toRowMatrix(); What you get is a matrix where the rows are arranged in whatever order they

Re: TallSkinnyQR

So b = 0.89 0.42 0.0 0.88 0.97 The solution at the bottom is the solution to Ax = b solved using Gaussian elimination. I guess another question is, is there another way to solve this problem? I'm trying to solve the least squares fit with a huge A (5MM x 1MM) x = inverse(A-transpose*A)*A-transose

Re: TallSkinnyQR

Hi Sean, Here you go: sparsematrix.txt = row, col ,val 0,0,.42 0,1,.28 0,2,.89 1,0,.83 1,1,.34 1,2,.42 2,0,.23 3,0,.42 3,1,.98 3,2,.88 4,0,.23 4,1,.36 4,2,.97 The vector is just the third column of the matrix which should give the trivial solution of [0,0,1] This translates to this which is cor

Re: TallSkinnyQR

Rather than post a large section of code, please post a small example of the input matrix and its decomposition, to illustrate what you're saying is out of order. On Tue, Nov 8, 2016 at 3:50 AM im281 wrote: > I am getting the correct rows but they are out of order. Is this a bug or > am > I doin