Thomas S. Dye <t...@tsdye.com> wrote: > Hi John, > > Have you tried wrapping your R read in system.time()? If you are right > about :results silent eating up lots of time, then this should fix the > problem. system.time yields just a bit of output, so shouldn't slow > things down if writing out the data is indeed the problem as you > suspect. >
I missed this in the thread so I thought I'd add one more thing to try: o run iostat (or equivalent) to check how your disks are doing. on linux, iostat is part of a package usually called sysstat. Nick > > > John Hendy <jw.he...@gmail.com> writes: > > > On Wed, Oct 31, 2012 at 3:12 PM, <cbe...@tajo.ucsd.edu> wrote: > > > >> John Hendy <jw.he...@gmail.com> writes: > >> > >> > On Wed, Oct 31, 2012 at 11:41 AM, <span dir="ltr"><mailto: > >> cbe...@tajo.ucsd.edu></span> wrote: > >> > John Hendy <mailto:jw.he...@gmail.com> writes: > >> > > >> >> I edited the subject to be more concise/clear.I let orgmode chug away > >> >> on reading in some ~10-30mb csv files for nearly 30min. > >> > > >> > [rest deleted] > >> > > >> > You need an ECM.I did my best to provide one, other than the file, which > >> I offered to provide if others requested that I upload it somewhere. Since > >> you have done so, so have I: > >> > - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM > >> > >> > Let me know if that doesn't work. I put it on Google docs and > >> sometimes have issues with the sharing settings... > >> > >> Not an ECM in my book, but ... > >> > >> > > What else would you like? I provided: > > - the config > > - the data > > - how to [attempt to] reproduce > > - the org-mode text > > > > > > > >> On my 4 year old MacBook: > >> > >> ,---- > >> | > >> | #+PROPERTY: session *R* > >> | > >> | #+name: bigcsv > >> | #+begin_src R > >> | bigcsv <- Sys.glob("~/Downloads/*.csv") > >> | #+end_src > >> | > >> | #+RESULTS: bigcsv > >> | : /Users/cberry/Downloads/test-file.csv > >> | > >> | #+name: readbig > >> | #+begin_src R :results output > >> | system.time( > >> | tmp <- read.csv(bigcsv) > >> | ) > >> | > >> | #+end_src > >> | > >> | #+RESULTS: readbig > >> | : user system elapsed > >> | : 5.679 0.306 6.002 > >> | > >> `---- > >> > >> About the same as running from ESS. > >> > >> > > Not sure what to say. Looking for ways to troubleshoot or confirm. Since > > you can't confirm, any suggestions on where I should look for my issue? I > > can't explain it! All I know is that org chugs and chugs and the direct > > execution in ESS session is lightning fast. > > > > > > > > FYI, > >> > >> > dim(tmp) > >> [1] 964805 9 > >> > >> > >> > > Was that just to show the size, or is something wrong with my file? > > > > > >> Chuck > >> > >> > > > > Thanks for attempting to replicate! > > John > > > > > >> [ rest deleted ] > >> > >> > >> > >> > > On Wed, Oct 31, 2012 at 3:12 PM, <cbe...@tajo.ucsd.edu> wrote: > > > > John Hendy <jw.he...@gmail.com> writes: > > > > > On Wed, Oct 31, 2012 at 11:41 AM, <span > > dir="ltr"><mailto:cbe...@tajo.ucsd.edu></span> wrote: > > > John Hendy <mailto:jw.he...@gmail.com> writes: > > > > > >> I edited the subject to be more concise/clear.I let orgmode > > chug away > > >> on reading in some ~10-30mb csv files for nearly 30min. > > > > > > [rest deleted] > > > > > > You need an ECM.I did my best to provide one, other than the > > file, which I offered to provide if others requested that I upload > > it somewhere. Since you have done so, so have I: > > > - https://docs.google.com/open?id=0BzQupOSnvw08WHdabHh5VVczRGM > > > > > Let me know if that doesn't work. I put it on Google docs > > and sometimes have issues with the sharing settings... > > > > Not an ECM in my book, but ... > > > > > > > > What else would you like? I provided: > > - the config > > - the data > > - how to [attempt to] reproduce > > - the org-mode text > > > > > > > > On my 4 year old MacBook: > > > > ,---- > > | > > | #+PROPERTY: session *R* > > | > > | #+name: bigcsv > > | #+begin_src R > > | bigcsv <- Sys.glob("~/Downloads/*.csv") > > | #+end_src > > | > > | #+RESULTS: bigcsv > > | : /Users/cberry/Downloads/test-file.csv > > | > > | #+name: readbig > > | #+begin_src R :results output > > | system.time( > > | tmp <- read.csv(bigcsv) > > | ) > > | > > | #+end_src > > | > > | #+RESULTS: readbig > > | : user system elapsed > > | : 5.679 0.306 6.002 > > | > > `---- > > > > About the same as running from ESS. > > > > > > > > Not sure what to say. Looking for ways to troubleshoot or confirm. > > Since you can't confirm, any suggestions on where I should look for my > > issue? I can't explain it! All I know is that org chugs and chugs and > > the direct execution in ESS session is lightning fast. > > > > > > > > FYI, > > > > > dim(tmp) > > [1] 964805 9 > > > > > > > > > > Was that just to show the size, or is something wrong with my file? > > > > > > Chuck > > > > > > > > > > Thanks for attempting to replicate! > > John > > > > > > [ rest deleted ] > > > > > > > > > > > > -- > Thomas S. Dye > http://www.tsdye.com >
