On Tue, 2013-01-01 at 20:01 +0000, Ken Moffat wrote: > On Tue, Jan 01, 2013 at 01:21:26PM -0600, Gerard Beekmans wrote: > > Hi guys, > > > > After reviewing logs I ended up having to block the wget user agent in > > Apache for the time being. Pages such as > > http://www.linuxfromscratch.org/lfs/downloads/stable/ are causing issues > > with wget. > > > > The name, last modified, size and description headers are clickable > > links to change the sorting of the page. This tricks wget's recursive > > mode into thinking they are different pages. > > Is this something we can change in the future, somewhere in the xml, > or is it another of those "we miss Manuel" moments ?
Nope, that page is served out by Apache using its autoindex module. Gerard, we could just configure Apache to use 'SuppressColumnSorting' (http://httpd.apache.org/docs/2.2/mod/mod_autoindex.html#indexoptions) - it won't stop bots from downloading masses of data if that's what they're intent on doing, but for otherwise innocent scripts that are being tripped up by the column sorting hyperlinks, it'll prevent them getting multiple copies of everything. Regards, Matt. -- http://linuxfromscratch.org/mailman/listinfo/lfs-dev FAQ: http://www.linuxfromscratch.org/faq/ Unsubscribe: See the above information page
