Jonas, do you have any performance and/or efficacy stats for your
URLRedirect plugin?
After months of near silence, I'm seeing an interesting (albeit
low volume) shortener campaign, that's picking up volume AND
effectiveness.
Only one of my 40-ish domains was getting these, then this week two
other domains started getting a trickle. They make up about 5.6%
of the post-Spamhaus volume for the main victim, and (currently)
much less than 1% for the two new victims.
They're mostly targeting BitLy, and they're getting MUCH better at
thwarting BitLy's detection (for a short period, it was near 100%
catch rate, now it's down to almost zero). The volume is low
enough that I've been batch HTTP-HEADing them, a few times each
day.
They're not doing anything stunningly innovative, and they "should"
be fairly obvious (both algorithmically and to the (experienced)
naked eye), but the volume must be small enough that they're
sneaking under the radar.
I've been killing these partly by giving BitLy/et-al a fairly hefty
score, UNLESS the sender uses the correct RealName in the To
header, or they've already got a generous skip rule.
That's brute force, and terribly inelegant.
The FP rate hasn't been "too" bad, and I'm of the opinion that a
URL shortener is an admission by the sender that the content is low
priority, and all but begs to be delayed and salvaged by our FP
pipeline.
Still, I'd like to find a more elegant way to handle these, partly
so I can be even more aggressive.
My main concern with doing real-time HTTP HEADs is performance.
Jonas, I've been thinking that if you embedded the SA spam score in
the HTTP request's Agent, that would provide BitLy/et-al with
extremely useful data, which should improve their detection rate
(if they choose to use the extra data).
You could also include the recipient's domain, which may help them
to correlate data. In the campaigns I've looked at, the patterns
are fairly obvious, as long as you look at _JUST_ the data for that
domain's spam (granted, I'm in a small domain environment, and that
really wouldn't be relevant for a big ISP's data, however that's
easy to separate at the shorteners' end).
Does anyone have a contact at BitLy? They seem to be on the ball,
and serious. They could well embrace cooperation. :)
I'm also wondering about using UDP to send a quick real-time,
no-response-needed message (instead of a high overhead HTTP
request), then (mostly) auto-quarantine, and later in a separate,
batch queue, do a proper HTTP Head. Anything that's clean after a
certain amount of time, could be automatically re-injected back
into the main queue. That would allow pooling of requests, and
shift the load from the main email gateway.
The UDP part would be pointless without cooperation from the
shortener services. If they do embrace it, they can use the UDP
data to more quickly identify most bad links, and have that all
ready for when we send out HTTP requests.
Ideally, if a pilot program proved successful, some of the
shorteners might be willing to host a proper DNS blocklist, which
would improve performance for everybody. :)
The commercial shorteners COULD even use a blocklist under their
control (i.e. preferred white/pass-listing) as a selling point for
their (future?) paying customers, and/or as an incentive for their
link creators to opt-in to a more rigorous creation channel
(i.e. still keep the no-brainer, Pakled-friendly channel, but also
have a channel for adults (in the original sense of that word) who
are willing to perform some level of verification).
I've also seeing a very pernicious Digg campaign, all being sent
via Hotmail. In some ways, that's more effective than the
mainstream shorteners, since it appears Digg does not check any
blocklists. :(
These services are just too dang tempting a target, so I expect
these campaigns to continue.
More fevered ramblings from one of your mostly harmless Iowa Geeks,
- "Chip"
