I'm just wondering are the SparkVector calculations really taking into
account the sparsity or just converting to dense?


On Fri, Apr 25, 2014 at 10:06 PM, John King <[email protected]>wrote:

> I've been trying to use the Naive Bayes classifier. Each example in the
> dataset is about 2 million features, only about 20-50 of which are
> non-zero, so the vectors are very sparse. I keep running out of memory
> though, even for about 1000 examples on 30gb RAM while the entire dataset
> is 4 million examples. And I would also like to note that I'm using the
> sparse vector class.
>

Reply via email to