Re: LinkWalker

2001-12-24 Thread Russell Coker
On Mon, 24 Dec 2001 06:42, Jeremy Lunn wrote: > On Sun, Dec 23, 2001 at 05:41:47PM +0100, Russell Coker wrote: > > I have a nasty web spider with an agent name of "LinkWalker" downloading > > everything on my site (including .tgz files). Does anyone know anything > > about it? > > Surely you'd be

Re: LinkWalker

2001-12-24 Thread Jeremy Lunn
On Mon, Dec 24, 2001 at 11:43:09AM +0100, Russell Coker wrote: > > > I have a nasty web spider with an agent name of "LinkWalker" downloading > > > everything on my site (including .tgz files). Does anyone know anything > > > about it? > > > > Surely you'd be able to disallow access to it with Ap

selam 2

2001-12-24 Thread ARZU COLAK
Selam sana bir site oneriyorum kesin bak! , OYUNLAR SADECE 2.750.000 TL! http://www.alisveris.sehri.com http://www.alisveris.sehri.com iyi gunler, Bu mesaj htp://www.aslan.mekani.com üzerinden yollanmistir! Uye olmak icin ; http://astavilla.kolayweb.com/haber.htm

Re: LinkWalker

2001-12-24 Thread Jeff Waugh
> > Why don't you just update your robots.txt to explicitly specify which > > files you don't or do, allow spiders access to. If it's a rule-obiding > > spider, that will be the end of it. > > I wasn't aware that there was any format to robots.txt, I thought that the > mere presense of such a

Re: LinkWalker

2001-12-24 Thread Russell Coker
On Mon, 24 Dec 2001 06:42, Jeremy Lunn wrote: > On Sun, Dec 23, 2001 at 05:41:47PM +0100, Russell Coker wrote: > > I have a nasty web spider with an agent name of "LinkWalker" downloading > > everything on my site (including .tgz files). Does anyone know anything > > about it? > > Surely you'd be

Re: LinkWalker

2001-12-24 Thread Jeremy Lunn
On Mon, Dec 24, 2001 at 11:43:09AM +0100, Russell Coker wrote: > > > I have a nasty web spider with an agent name of "LinkWalker" downloading > > > everything on my site (including .tgz files). Does anyone know anything > > > about it? > > > > Surely you'd be able to disallow access to it with Apa

Re: LinkWalker

2001-12-24 Thread Jeff Waugh
> > Why don't you just update your robots.txt to explicitly specify which > > files you don't or do, allow spiders access to. If it's a rule-obiding > > spider, that will be the end of it. > > I wasn't aware that there was any format to robots.txt, I thought that the > mere presense of such a f