On 13-04-26 3:00 PM, Kevin Hao wrote:
Hi Ye,
Thanks.
That is a good method. have any other methods instead of using database?
If you know the format of the file, you can probably write something in
C (or other language) that is faster than R. Convert your .csv file to
a nice binary format, and R will read it in no time at all.
If writing it in C is hard, then R is probably a better use of your
time. Read the file once, write it out using saveRDS(), and read it in
using readRDS() after that.
In either case, the secret is to do the conversion from ugly character
encoded numbers to beautiful binary numbers just once.
Duncan Murdoch
kevin
On Fri, Apr 26, 2013 at 1:58 PM, Ye Lin <ye...@lbl.gov> wrote:
Have you think of build a database then then let R read it thru that db
instead of your desktop?
On Fri, Apr 26, 2013 at 8:09 AM, Kevin Hao <rfans4ch...@gmail.com> wrote:
Hi all scientists,
Recently, I am dealing with big data ( >3G txt or csv format ) in my
desktop (windows 7 - 64 bit version), but I can not read them faster,
thought I search from internet. [define colClasses for read.table, cobycol
and limma packages I have use them, but it is not so fast].
Could you share your methods to read big data to R faster?
Though this is an odd question, but we need it really.
Any suggest appreciates.
Thank you very much.
kevin
[[alternative HTML version deleted]]
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[[alternative HTML version deleted]]
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
