I'm just wondering are the SparkVector calculations really taking into account the sparsity or just converting to dense?
On Fri, Apr 25, 2014 at 10:06 PM, John King <[email protected]>wrote: > I've been trying to use the Naive Bayes classifier. Each example in the > dataset is about 2 million features, only about 20-50 of which are > non-zero, so the vectors are very sparse. I keep running out of memory > though, even for about 1000 examples on 30gb RAM while the entire dataset > is 4 million examples. And I would also like to note that I'm using the > sparse vector class. >
