Which version of mllib are you using? For Spark 1.0, mllib will
support sparse feature vector which will improve performance a lot
when computing the distance between points and centroid.

Sincerely,

DB Tsai
-------------------------------------------------------
My Blog: https://www.dbtsai.com
LinkedIn: https://www.linkedin.com/in/dbtsai


On Sat, Apr 26, 2014 at 5:49 AM, John King <[email protected]> wrote:
> I'm just wondering are the SparkVector calculations really taking into
> account the sparsity or just converting to dense?
>
>
> On Fri, Apr 25, 2014 at 10:06 PM, John King <[email protected]>
> wrote:
>>
>> I've been trying to use the Naive Bayes classifier. Each example in the
>> dataset is about 2 million features, only about 20-50 of which are non-zero,
>> so the vectors are very sparse. I keep running out of memory though, even
>> for about 1000 examples on 30gb RAM while the entire dataset is 4 million
>> examples. And I would also like to note that I'm using the sparse vector
>> class.
>
>

Reply via email to