I am trying to perform Singular Value Decomposition (SVD) on a Term Document Matrix I created using the 'tm' package. Eventually I want to do a Latent Semantic Analysis (LSA).
There are 5677 documents with 771 terms (the DTM is 771 x 5677). When I try to do the SVD, it runs out of memory. I am using a 12GB Dual core Machine with Windows XP and don't think I can increase the memory anymore. Are there any other memory efficient methods to find the SVD? The term document is obtained using: tdm2 <- TermDocumentMatrix(tr1,control=list(weighting=weightTf,minWordLength=3)) str(tdm2) List of 6 $ i : int [1:6438] 202 729 737 278 402 621 654 718 157 380 ... $ j : int [1:6438] 1 2 3 7 7 7 7 8 10 10 ... $ v : num [1:6438] 8 5 6 9 5 7 5 6 5 7 ... $ nrow : int 771 $ ncol : int 5677 $ dimnames:List of 2 ..$ Terms: chr [1:771] "access" "accessori" "accumul" "acoust" ... ..$ Docs : chr [1:5677] "1" "2" "3" "4" ... - attr(*, "class")= chr [1:2] "TermDocumentMatrix" "simple_triplet_matrix" - attr(*, "Weighting")= chr [1:2] "term frequency" "tf" SVD is calcualted using: > tdm_matrix <- as.matrix(tdm2) > svd_out<-svd(tdm_matrix) Error: cannot allocate vector of size 767.7 Mb In addition: Warning messages: 1: In matrix(0, n, np) : Reached total allocation of 3583Mb: see help(memory.size) 2: In matrix(0, n, np) : Reached total allocation of 3583Mb: see help(memory.size) 3: In matrix(0, n, np) : Reached total allocation of 3583Mb: see help(memory.size) 4: In matrix(0, n, np) : Reached total allocation of 3583Mb: see help(memory.size) Thank you. Ravi -- View this message in context: http://r.789695.n4.nabble.com/SVD-Memory-Issue-tp3809667p3809667.html Sent from the R help mailing list archive at Nabble.com. ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
