Wget ignores robot.txt entry

[L Anderson]

Using the latest of things Cygwin, I downloaded some stuff with wget from <http://cygwin.com> to peruse off-line and noticed a problem I can't explain:

The <http://cygwin.com/robots.txt> file has the entries:

User-agent: *
Disallow: /snapshots/
Disallow: /cgi-bin/
Disallow: /cgi2-bin/

so wget should not download /cgi-bin/.

However, "wget -o cygwincom.log -m -p --no-parent -X /cygwin,/ml http://cygwin.com/"; downloads /cgi-bin anyway.

NB. "wget -o cygwincom.log -m -p --no-parent -X /cgi-bin,/cygwin,/ml http://cygwin.com/ doesn't download /cgi-bin

I ran a validity check on <http://cygwin.com/robots.txt> and found no errors.

Is this a bug in wget or am I doing something wrong?

Thanks,

Lowell Anderson



--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Bug reporting: http://cygwin.com/bugs.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/