You seem to want your cake and eat it too. Not unexpected, but you may have
your work cut out to learn about the price of having it all.
Plotting: pretty silly to stick with gigabytes of data in your plots. Some kind
of aggregation seems required here, with the raw data being a stepping stone to
that goal.
Loading: if you don't have RAM, buy more or use one of the disk-based
solutions. There are proprietary solutions for a fee, and there are packages
like ff. When I have dealt with large data sets I have used sqldf or RODBC
(which I think works best for read-only access), so I cannot advise you on ff.
---------------------------------------------------------------------------
Jeff Newmiller The ..... ..... Go Live...
DCN:<jdnew...@dcn.davis.ca.us> Basics: ##.#. ##.#. Live Go...
Live: OO#.. Dead: OO#.. Playing
Research Engineer (Solar/Batteries O.O#. #.O#. with
/Software/Embedded Controllers) .OO#. .OO#. rocks...1k
---------------------------------------------------------------------------
Sent from my phone. Please excuse my brevity.
On July 14, 2015 3:21:42 PM PDT, "Dupuis, Robert" <dup...@beaconpower.com>
wrote:
>I'm relatively new to using R, and I am trying to find a decent
>solution for my current dilemma.
>
>Right now, I am currently trying to parse second data from a 7 months
>of CSV data. This is over 10GB of data, and I've run into some "memory
>issues" loading them all into a single dataset to be plotted. If
>possible, I'd really like to keep both the one second resolution, and
>all 100 or so columns intact to make things easier on myself.
>
>The problem I have is that the machine that is running this script only
>has 8GB of RAM. I've had issues parsing files with lapply, and some
>sort of csv reader. So far I've tried read.csv, readr.read_table, and
>data.table.fread with only fread having any sort of memory management
>(fread seems to crash on me however). The basic approach I am using is
>as follows:
>
># Get the data
>files = list.files(pattern="*.csv")
>set <- lapply(files, function(x) fread(x, header = T, sep = ','))
>#replace fread with something that can parse csv data
>
># Handle the data (Do my plotting down here)
>...
>
>These processes work with smaller data sets, but I would like to in a
>worse case scenario be able to parse through 1 year data which would be
>around 20GB.
>
>Thank you for your time,
>Robert Dupuis
>
>______________________________________________
>R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.
______________________________________________
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
