On 5/6/19 2:27 PM, Stephen Berman wrote:
On Mon, 6 May 2019 11:12:25 +0200 Ralf Stubner <ralf.stub...@daqana.com> wrote:
On 04.05.19 19:04, Stephen Berman wrote:
In versions of R prior to 3.6.0 the following invocation succeeds,
returning the data frame shown:
read.table("https://www.dwds.de/r/stat?corpus=kern&cnt=tokens&date=decade&format=text";,
header=TRUE)
Dekade Anzahl
1 1900 11467254
2 1910 13023370
3 1920 13434601
4 1930 13296355
5 1940 12121250
6 1950 13191131
7 1960 10587420
8 1970 10944129
9 1980 11279439
10 1990 12052652
But in version 3.6.0 it fails:
read.table("https://www.dwds.de/r/stat?corpus=kern&cnt=tokens&date=decade&format=text";,
header=TRUE)
Error in file(file, "rt") :
cannot open the connection to
'https://www.dwds.de/r/stat?corpus=kern&cnt=tokens&date=decade&format=text'
In addition: Warning message:
In file(file, "rt") :
cannot open URL
'https://www.dwds.de/r/stat?corpus=kern&cnt=tokens&date=decade&format=text':
HTTP status was '403 Forbidden'
I can reproduce the behavior on Debian using the CRAN supplied package
for R 3.6.0. Trying to read the page with 'curl' produces also a 403
error plus some HTML text (in German) explaining that I am treated as a
'robot' due to the supplied User-Agent (here: curl/7.52.1). One
suggested solution is to adjust that value which does solve the issue:
> options(HTTPUserAgent='mozilla')
I confirm that works for me, too. Thanks! FWIW, the default value of
HTTPUserAgent in R 3.6 here is "R (3.6.0 x86_64-suse-linux-gnu x86_64
linux-gnu)", and using this (in R 3.6) fails as I reported, while the
default value of HTTPUserAgent in R 3.5 here is "R (3.5.0
x86_64-suse-linux-gnu x86_64 linux-gnu)" and using that (in R 3.5)
succeeds. However, setting HTTPUserAgent in R 3.5 to "libcurl/7.60.0"
fails just as it does in 3.6. It's not clear to me if this particular
website is being too restrictive or if R 3.6 should deal with it, or at
least mention the issue in NEWS or somewhere else.
This is because (from NEWS:)
The default ‘user agent’ has been changed when accessing http://
and https:// sites using libcurl. (A site was found which caused
libcurl to infinite-loop with the previous default.)
This website is ok with the default R user agent specification (also for
R 3.6 and R-devel), but it is not ok with "libcurl/...". Setting the
user agent to anything starting with "R (" will not help in R 3.6,
because it will get automatically changed to "libcurl/..." when libcurl
is used (note using wget and curl on the command line fails on this
website). I am afraid it has to be solved on the user side (e.g. as
hinted in that German text one gets when requesting the page using curl)
- R should not attempt to circumvent access restrictions on external
websites.
Best
Tomas
Steve Berman
______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
