That is what I feared. I know other people on slow connections have done this without issue ( at least they didnt report an issue ) I had a similar issue with geonames.org who at least published their terms of service (requests per second or sumptin like that.. so could program around it) I'll hunt around on their ftp and then write the admin a note.. i really don't want to brute force the matter
it's government data made available for the public so I expect the admin will be helpful. Thanks for confirming what I suspected, for a minute [ .ed: for two days] I thought I had taken crazy pills. i did note, however, some odd behavior with tryCatch, where statements after the "finally={} were executed. Not sure if that deserves a bug report. On Tue, Sep 21, 2010 at 2:33 AM, Barry Rowlingson < b.rowling...@lancaster.ac.uk> wrote: > On Tue, Sep 21, 2010 at 9:39 AM, steven mosher <mosherste...@gmail.com> > wrote: > > I'm accessing around 95 tar files on an FTP server ranging in size > between > > 10 and 40MB a piece. > > > > while certainly can click on them and download them outside of R, I'd > like > > to have my script do it. > > > > Retrieving the ftp directory with RCurl works fine (about 90% of the > time) > > > > but downloading the files by looping through all the files is a random > > process. > > > > I may get 1-8 files download and then it throws an error > > > > "cannot open URL " > > > > sometimes I only can get 1 file before this error. with tryCatch() I've > been > > able to do some clean up > > after the crash, but automating this whole download process has turned > into > > a bit of a circus. > > > > The parameters (url, destfile, mode) are all correct in the download.file > > call as the second attempt at a url will often succeed. > > > > Is there anyway to get a deeper look at the cause of the problem? I've > tried > > closing all connections > > in between each download. any pointers would be welcomed. > > Sounds to me like the FTP server is operating some kind of rate > limiting. Do you have access to the server log files, or the server > administrator, or perhaps the server's terms and conditions to see if > its so :) > > Barry > [[alternative HTML version deleted]] ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.