I have a version that uses bigmemory on my blog, but looks at distance on a sphere for a 36k * 36K matrix
not hundreds of Gb so I dont know if the approach will work for you http://stevemosher.wordpress.com/2012/04/12/nick-stokes-distance-code-now-with-big-memory/ Steve However, I never tested it with On May 2, 2013 9:40 PM, "HJ YAN" <yhj...@googlemail.com> wrote: > Dear R users > > > I wondered if any of you ever tried to calculate distance matrix with very > large data set, and if anyone out there can confirm this error message I > got actually mean that my data is too large for this task. > > negative length vectors are not allowed > > > My data size and code used > > dim(mydata_nor)[1] 365000 144> d <- dist(mydata_nor, method = > "euclidean") > > > > Here my data has 1000 samples each has a year data observed by 10 minutes > interval daily, so the size is (365* 1000) * 144. > > > I checked the manual of function 'dist' but can not see the upper limit > size allowed, and I bet there should be one, so any hints is appreciated. > > > I would also be grateful if any other method for calculating distance > matrix for large dataset could be advised. > > > > I appreciate reproducible code should be provided for your advice, so try > below if needed: > > A<-matrix(1:365000*144,nrow=365000,ncol=144)> dim(A)[1] 365000 144> > d1<-dist(A,method="euclidean")Error in dist(A, method = "euclidean") : > negative length vectors are not allowed > > > > > Many thanks in advance! > > HJ > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > [[alternative HTML version deleted]] ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
