Thanks guys, writing smaller pieces did the trick. Still takes about 4 hours
to do but that more manageable than over 30 hours without even writing a
line.
jholtman wrote:
>
> What do you want to export it to and in what format? You will probably
> have
> a problem trying to write that large an object out at one time. Try
> instead
> opening a connecting and the writing much smaller pieces to the file:
>
> x <- matrix(runif(10000), 100)
> x[,1] <- 1:100 # put some labels for testing
> x.out <- file('/test.csv', 'w')
> # not efficient, just for demonstration
> for (i in 1:100) write.table(x[i,, drop=FALSE], file=x.out,
> sep=',', col.names=FALSE, row.names=FALSE)
> close(x.out)
>
>
>
> On Mon, Jun 29, 2009 at 2:51 AM, DrDimo
> <dmitri.moura...@ludwig.edu.au>wrote:
>
>>
>> Hi all,
>>
>> I am currently working on a particularly large (data wise) project, the
>> problem I am having is with exporting the data (for use later on). My
>> current sample dataset contains about 400million entries (approximately
>> 20000 columns x 20000 rows). I tried using the write.table command, but
>> 8
>> hours later its still going. I will be trying write.matrix (MASS) next as
>> it
>> is supposed to handle things a bit quicker, but I thought I would ask if
>> anyone has had experience with exporting large datasets and what is the
>> quickest method to do so.
>>
>> Thanks in advance
>>
>> -Dmitri
>> --
>> View this message in context:
>> http://www.nabble.com/Exporting-large-datasets-tp24249149p24249149.html
>> Sent from the R help mailing list archive at Nabble.com.
>>
>> ______________________________________________
>> R-help@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html<http://www.r-project.org/posting-guide.html>
>> and provide commented, minimal, self-contained, reproducible code.
>>
>
>
>
> --
> Jim Holtman
> Cincinnati, OH
> +1 513 646 9390
>
> What is the problem that you are trying to solve?
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
>
--
View this message in context:
http://www.nabble.com/Exporting-large-datasets-tp24249149p24299303.html
Sent from the R help mailing list archive at Nabble.com.
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
