On Sat, Jan 1, 2011 at 7:19 AM, Steve Freegard <st...@stevefreegard.com>wrote:
> 7) How fast are typical URL shortening responses? What is the timeout? We
> want to avoid degrading the scan time and delivery performance of
> spamassassin, but in a way that cannot be abused by the spammer to evade
> detection.
>
>
> This could be a problem with your huge list of shortening services. If you
> blindly include all possible shortening services, spammers could
> purposefully use only the slowest in order timeout spamassasin. Web
> browsers are more forgiving in timeouts, so a slow redirector is the ideal
> way to evade your plugin.
>
> It is possible that you may want to include only the most reputable
> shortening services by default, because you don't know what will happen
> during the multiple years of your plugin being deployed on arbitrary
> servers. Other less reputable shortening services might be hijacked, domain
> ownership changed, or simply neglected and become slow. Such services may
> need to be blacklisted entirely. For the non-default shortening services,
> it may be safe only if it can be updated via sa-update.
>
>
> The timeout is set to 5 seconds and with a default of 10 short URIs scanned
> it would take 50 seconds before it timed out the lookups. Thinking about it
> I could possibly mitigate this by tracking timeouts by shortener domain; so
> if the 1st lookup to that shortener service timed-out then it wouldn't
> attempt the rest.
>
Everything else about this sounds very good, but this part is a bit
worrisome. Looking through my logs, my average scantime is under 1 second.
During debugging a timeout of 5 seconds would be fine in order to help
determine how fast the shorteners typically respond. But changes are needed
to avoid severely impacting delivery times.
* Consecutive timeouts wont work. The combined timeout of all short lookups
when this plugin goes into production must be under maybe 3-5 seconds.
* I know this would be difficult, but would it be possible to make
asynchronous and concurrent queries to the shorteners instead of
one-after-another? Kind of like how the URIDNSBL plugin currently works.
There might be some complications here, like most HTTP servers will only
respond to the first two concurrent connections from an IP address while
further connections are serviced only after the first two have disconnected.
Rule Ideas
========
SHORT_URL_MULTI10
SHORT_URL_TOOMANY
Rules triggering on suspicious behavior even if your plugin didn't have time
to query it all.
SHORT_URL_TIMEOUT
The plugin could print out which URL timed out. Something like:
X-Spam-Report:
* 0.5 SHORT_URL_TIMEOUT Shortened URL Timedout
* [3 second timeout for http://example.com/298fauu]
Warren
