That is what I feared. I know other people on slow connections have done
this without issue ( at least they didnt report an issue ) I had a similar
issue with geonames.org who at least published their terms
of service (requests per second or sumptin like that.. so could program
around it) I'll hunt around
on their ftp and then write the admin a note.. i really don't want to brute
force the matter
it's government data made available for the public so I expect the admin
will be helpful.
Thanks for confirming what I suspected, for a minute [ .ed: for two days] I
thought I had taken crazy pills.
i did note, however, some odd behavior with tryCatch, where statements
after the "finally={} were executed. Not sure if that deserves a bug report.
On Tue, Sep 21, 2010 at 2:33 AM, Barry Rowlingson <
b.rowling...@lancaster.ac.uk> wrote:
> On Tue, Sep 21, 2010 at 9:39 AM, steven mosher <mosherste...@gmail.com>
> wrote:
> > I'm accessing around 95 tar files on an FTP server ranging in size
> between
> > 10 and 40MB a piece.
> >
> > while certainly can click on them and download them outside of R, I'd
> like
> > to have my script do it.
> >
> > Retrieving the ftp directory with RCurl works fine (about 90% of the
> time)
> >
> > but downloading the files by looping through all the files is a random
> > process.
> >
> > I may get 1-8 files download and then it throws an error
> >
> > "cannot open URL "
> >
> > sometimes I only can get 1 file before this error. with tryCatch() I've
> been
> > able to do some clean up
> > after the crash, but automating this whole download process has turned
> into
> > a bit of a circus.
> >
> > The parameters (url, destfile, mode) are all correct in the download.file
> > call as the second attempt at a url will often succeed.
> >
> > Is there anyway to get a deeper look at the cause of the problem? I've
> tried
> > closing all connections
> > in between each download. any pointers would be welcomed.
>
> Sounds to me like the FTP server is operating some kind of rate
> limiting. Do you have access to the server log files, or the server
> administrator, or perhaps the server's terms and conditions to see if
> its so :)
>
> Barry
>
[[alternative HTML version deleted]]
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
