2018-02-26 10:41 GMT-03:00 Dianne Skoll <d...@roaringpenguin.com>:
> On Mon, 26 Feb 2018 00:07:54 -0600 (CST)
> David B Funk <dbf...@engineering.uiowa.edu> wrote:
>
> > So my bet is that the spammers are crafty enough to check things like
> > browser referrer, cookies, etc to detect/differentiate a browser vs a
> > link-checker.
>
> Yep. You need to fake your User-Agent (not hard) and put a limit on
> the number of redirections you follow or you can end up in trouble.
>
> I picked a likely-looking agent out of our logs and use this:
>
> $ua->agent('Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64;
> Trident/7.0; SLCC2;');
>
> Might need revising later because it's pretty ancient!
>
That is right Dianne!
Our script is using "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0)
Gecko/20100101 Firefox/57.0":
https://www.dropbox.com/s/5aorrijafw5ygk0/uribl.pl?dl=0
This user agent was most used, some time ago, according to this site:
https://techblog.willshouse.com/2012/01/03/most-common-user-agents/
This is out of date now, than you can update it if you want. We will do it
soon.
>
> Regards,
>
> Dianne.
>
