O, wait a sec - does it mean I can't feed my objects into sql commands?
On Sat, Oct 23, 2010 at 10:07 AM, Dimitri Liakhovitski
<dimitri.liakhovit...@gmail.com> wrote:
> I just tried it:
>
> for(i in 11:16){ #i<-11
> start<-Sys.time()
> print(start)
> flush.console()
> filename<-paste("skipped millions- ",i,".txt",sep="")
> mydata<-read.csv.sql("myfilel.txt", sep="|", eol="\r\n", sql =
> "select * from file limit 1000000, (1000000*i-1)")
> write.table(mydata,file=filename,sep="\t",row.names=F)
> end<-Sys.time()
> print(end-start)
> flush.console()
> }
>
> It started running at 9:52 am.
> Around 10:05 am I got this error:
>
> Error in sqliteExecStatement(con, statement, bind.data) :
> RS-DBI driver: (error in statement: no such column: i)
>
> What does it mean?
> Thank you!
> Dimitri
>
> On Sat, Oct 23, 2010 at 9:44 AM, Dimitri Liakhovitski
> <dimitri.liakhovit...@gmail.com> wrote:
>> Oh, I understand - I did not realize it's reading in the whole file.
>> So, is there any way to make it read it in only once and the spit into
>> R just one piece (e.g., 1 million rows), write a regular file out
>> (e.g., a txt using write.table), and then grab the next million?
>> Because I was planning to do something like this (I have 17+ million rows):
>>
>> for(1:16){
>> filename<-paste("million number ",i,".txt",sep="")
>> mydata<-read.csv.sql("myfile.txt", sep="|", eol="\r\n", sql = "select
>> * from file limit 1000000, (1000000*i-1)")
>> write.table(mydata,file=filename,sep="\t",row.names=F)
>> }
>>
>> But if each iteration it will be reading in the whole file for a long
>> time - it'll take a long time... Is there any way to make it read the
>> file in only once? I guess not - because there is not enough memory to
>> hold it in the first place...
>> Thanks again for your advice!
>> Dimitri
>>
>>
>>
>> On Sat, Oct 23, 2010 at 9:32 AM, Gabor Grothendieck
>> <ggrothendi...@gmail.com> wrote:
>>> On Sat, Oct 23, 2010 at 9:20 AM, Dimitri Liakhovitski
>>> <dimitri.liakhovit...@gmail.com> wrote:
>>>> This is very helpful, Gabor.
>>>> I've run the code to figure out the end of the line and here is what I
>>>> am seeing at the end of each line: \r\n
>>>> So, I specified like this:
>>>> mydata<-read.csv.sql("myfile.txt", sep="|", eol="\r\n", sql = "select
>>>> * from file limit 200, 100")
>>>>
>>>> However, again it's hanging again. Another typo?
>>>>
>>>
>>> I wonder if its just taking longer than you think. It does read the
>>> entire file into sqlite even if you only read a portion from sqlite to
>>> R so if the file is very long it will still take some time. Try
>>> creating a small file of a few hundred lines from your file and
>>> experiment on that until you get it working.
>>>
>>> --
>>> Statistics & Software Consulting
>>> GKX Group, GKX Associates Inc.
>>> tel: 1-877-GKX-GROUP
>>> email: ggrothendieck at gmail.com
>>>
>>
>>
>>
>> --
>> Dimitri Liakhovitski
>> Ninah Consulting
>> www.ninah.com
>>
>
>
>
> --
> Dimitri Liakhovitski
> Ninah Consulting
> www.ninah.com
>
--
Dimitri Liakhovitski
Ninah Consulting
www.ninah.com
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
