Thank you, it is exactly that. I have followed your idea of chunks (1 GB chunks, on the safe side), and appended them. Worked like charm, thank you.
--Maxime From: "Stefan Evert (Mailing Lists)" <stefa...@collocations.de<mailto:stefa...@collocations.de>> Date: mercredi 17 septembre 2014 15:39 To: Maxime Vall�e <vall...@iarc.fr<mailto:vall...@iarc.fr>> Cc: R-help Mailing List <r-help@r-project.org<mailto:r-help@r-project.org>> Subject: Re: [R] R "write" strange behavior in huge file You probably told R to write out the file as a single long line with fields separated alternately by 380 TABs and one newline � that�s what the ncol argument does (write is just a small wrapper around cat()). cat() doesn�t print lines that are longer than 2 GiB, so it will insert an extra \n after every 2 GiB of data. (IIRC, this is because in the C code, fill=FALSE is replaced by fill=MAX_INT or so.) The only way around this limitation that I can think of is to write a wrapper function that breaks up the matrix or list of vectors in smaller chunks and appends them separately to the output file. I�m planning to add such a function to one of my packages, so I�d be interested if somebody has a better solution. Best, Stefan On 16 Sep 2014, at 18:54, Maxime Vallee <vall...@iarc.fr<mailto:vall...@iarc.fr>> wrote: In my script I have one list of 1,132,533 vectors (each vector contains 381 elements). When I use "write" to save this list in a flat text file (I unlist my list, separate by tabs, and set ncol to 381), I end up with a file of 1,132,535 lines (2 additional lines). I checked back, my R list do not have those two additional items before writing. With awk, I determined if lines where not made of 381 fields: there were two, separated by around 400k lines. I made sub-files, using those "incomplete" lines as boundaries. My files are very close in size : 1.9 GB (respectively 1971841853 B and 1972614897 B). It feels like a 32 bit / 64 bit issue. My R version is this: ./Rscript -e 'sessionInfo()$platform' [1] "x86_64-unknown-linux-gnu (64-bit)" There is somewhere, reaching 1.9 GB, something that is changing my tabs to unwanted carriage returns... Any idea that might cause this, and if it looks solvable in R? ----------------------------------------------------------------------- This message and its attachments are strictly confidenti...{{dropped:11}}
______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
