thank you all very much. Kevin
On Sat, Apr 27, 2013 at 11:51 AM, Jan van der Laan <rh...@eoos.dds.nl>wrote: > > I believe it was already mentioned, but I can recommend the LaF package > (not completely impartial being the maintainer of LaF ;-) > > However, the speed differences between packages will not be very large. > Eventually all packages will have to read in 6 GB of data and convert the > text data to numeric data. So the tricks are to > 1 only read in columns that you need > 2 only read in lines that you need > 3 and if you need to read the data more than once convert it to some > binary format first (RDS, ff, sqlite, bigmemory, ...). Most packages have > routines to convert CSV files to the binary format. > > With all of the above LaF helps. ffbase contains a routine laf_to_ffdf to > convert to to ff format. > > > HTH, > > Jan > > > > > On 04/27/2013 04:34 AM, Kevin Hao wrote: > >> Thank you very much. >> >> More and more methods are coming. That sounds great! >> >> >> Thanks, >> >> kevin >> >> >> >> On Fri, Apr 26, 2013 at 7:51 PM, Duncan Murdoch <murdoch.dun...@gmail.com >> >**wrote: >> >> On 13-04-26 3:00 PM, Kevin Hao wrote: >>> >>> Hi Ye, >>>> >>>> Thanks. >>>> >>>> That is a good method. have any other methods instead of using database? >>>> >>>> >>> If you know the format of the file, you can probably write something in C >>> (or other language) that is faster than R. Convert your .csv file to a >>> nice binary format, and R will read it in no time at all. >>> >>> If writing it in C is hard, then R is probably a better use of your time. >>> Read the file once, write it out using saveRDS(), and read it in using >>> readRDS() after that. >>> >>> In either case, the secret is to do the conversion from ugly character >>> encoded numbers to beautiful binary numbers just once. >>> >>> Duncan Murdoch >>> >>> >>> >>> kevin >>>> >>>> >>>> On Fri, Apr 26, 2013 at 1:58 PM, Ye Lin <ye...@lbl.gov> wrote: >>>> >>>> Have you think of build a database then then let R read it thru that >>>> db >>>> >>>>> instead of your desktop? >>>>> >>>>> >>>>> On Fri, Apr 26, 2013 at 8:09 AM, Kevin Hao <rfans4ch...@gmail.com> >>>>> wrote: >>>>> >>>>> Hi all scientists, >>>>> >>>>>> >>>>>> Recently, I am dealing with big data ( >3G txt or csv format ) in my >>>>>> desktop (windows 7 - 64 bit version), but I can not read them faster, >>>>>> thought I search from internet. [define colClasses for read.table, >>>>>> cobycol >>>>>> and limma packages I have use them, but it is not so fast]. >>>>>> >>>>>> Could you share your methods to read big data to R faster? >>>>>> >>>>>> Though this is an odd question, but we need it really. >>>>>> >>>>>> Any suggest appreciates. >>>>>> >>>>>> Thank you very much. >>>>>> >>>>>> >>>>>> kevin >>>>>> >>>>>> [[alternative HTML version deleted]] >>>>>> >>>>>> ______________________________****________________ >>>>>> R-help@r-project.org mailing list >>>>>> https://stat.ethz.ch/mailman/****listinfo/r-help<https://stat.ethz.ch/mailman/**listinfo/r-help> >>>>>> <https://stat.**ethz.ch/mailman/listinfo/r-**help<https://stat.ethz.ch/mailman/listinfo/r-help> >>>>>> > >>>>>> >>>>>> PLEASE do read the posting guide >>>>>> http://www.R-project.org/****posting-guide.html<http://www.R-project.org/**posting-guide.html> >>>>>> <http://www.**R-project.org/posting-guide.**html<http://www.R-project.org/posting-guide.html> >>>>>> > >>>>>> >>>>>> and provide commented, minimal, self-contained, reproducible code. >>>>>> >>>>>> >>>>>> >>>>> >>>>> [[alternative HTML version deleted]] >>>> >>>> ______________________________****________________ >>>> R-help@r-project.org mailing list >>>> https://stat.ethz.ch/mailman/****listinfo/r-help<https://stat.ethz.ch/mailman/**listinfo/r-help> >>>> <https://stat.**ethz.ch/mailman/listinfo/r-**help<https://stat.ethz.ch/mailman/listinfo/r-help> >>>> > >>>> PLEASE do read the posting guide http://www.R-project.org/** >>>> posting-guide.html >>>> <http://www.R-project.org/**posting-guide.html<http://www.R-project.org/posting-guide.html> >>>> > >>>> >>>> and provide commented, minimal, self-contained, reproducible code. >>>> >>>> >>>> >>> >> [[alternative HTML version deleted]] >> >> ______________________________**________________ >> R-help@r-project.org mailing list >> https://stat.ethz.ch/mailman/**listinfo/r-help<https://stat.ethz.ch/mailman/listinfo/r-help> >> PLEASE do read the posting guide http://www.R-project.org/** >> posting-guide.html <http://www.R-project.org/posting-guide.html> >> and provide commented, minimal, self-contained, reproducible code. >> >> [[alternative HTML version deleted]] ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
