Some have requested my scripts and configurations so here it is. Below
you fill find the spamd-dnsbl and spamclusterd scripts that are used for
blacklisting spammers and whitelisting networks, respectively. Also
included is dnsbl-check which I use for testing IPs against multiple DNSBLs.
In the crontab below, you will see that I archive the spamdb daily and
save some stats mainly for post analysis. For instance, my initial spam
fighting technique many years ago (prior to enabling spamd actually) was
to block the IP networks (20,000+ IPv4 networks) of the countries in
which we received the most spam, yet weren't expecting legitimate email
from (i.e. China, Russia, India, Brazil, etc.). I still had this enabled
up until 2016-12-17. So I make notes of changes like this to see the
positive or negative effects and I have the spamdb archives to assist
the analysis. Changing spamd_flags is something else I document.
A side note: Years ago, blocking spamming countries, for me here in the
US, essentially got rid of my spam problem, but has become ineffective
as many spammers are sending from US networks now, thus spamd. It has
only been three days since I disabled spam country blocking, but I have
received exactly 2 emails that have made it pass spamd, which would have
otherwise been blocked by the country IP block. Not bad, but we'll see
what the stats look like in a couple of weeks. However, I can guarantee
that the number of trapped entries in my spamdb will increase. I
originally created my pf table of spamming countries from
http://www.ipdeny.com/ipblocks/data/countries/
One of the other tests, which had significant impact, was using
spamd.alloweddomains. I tried a few things, but settled on my current
setup: for one email domain I list just the domain part (e.g.
@domain1.com), but for the other domain, which has limited users, I list
the full email addresses of all current accounts (e.g.
us...@domain2.com, us...@domain2.com, ...). This increased my TRAPPED
entries by 30%. These additional TRAPPED IPs were mainly one-shot
spammers, so it was nice to tarpit them while I had the chance. So far
spamd has been very effective so I haven't defined and published any
SPAMTRAP addresses, but this is just another knob I can turn on and
measure if needed.
To assist with spam management without root privileges, I added the spam
administrator to the _spamd group, gave r/w group privileges on
/var/db/spamd, and added a few pfctl commands to the doas.conf.
Overall I am ecstatic about spamd and its integration with pf, as well
as the simple spamdb interface (with the help of grep(1), cut(1),
sort(1), wc(1), column(1), sed(1), etc.). It is an extremely flexible
and powerful toolset. Hopefully my experience and scripts are helpful to
other spam fighters. I think you can look to other projects, like
spamassassin for example, to get ideas of spam fighting techniques which
can be implemented at a lower level using pf and spamd. For example, a
set of factors could determine a spam "score" similar to spamassassin:
if an IP is on multiple DNSBLs (each list weighted by quality), the DNS
PTR doesn't correspond to the HELO, and it fails SPF, then it is
probably safe to blacklist. The bgp-spamd.net project is another tool
that could be added to the mix. You will have to balance complexity and
effectiveness, but I would encourage simplicity and minimal resource usage.
Again, hats off to all the developers.
=== spamclusterd ===
#!/bin/sh
#
# Whitelist an SMTP cluster network.
#
# NOTE: pipe spamdb(8) or an archive to stdin.
extract_helo_tld() { echo "$1" | sed -En 's/.*[[:<:]]([^.]+\.[^.]+)$/\1/p'; }
extract_ip_net() { echo "${1%.*}"; }
print_ip_net_with_mask() {
echo "$(extract_ip_net $1).0/24"
}
helo_tld_match()
{
tld1=$(extract_helo_tld "$1")
tld2=$(extract_helo_tld "$2")
[[ -n $tld1 && $tld1 = $tld2 ]]
}
ip_net_match()
{
net1=$(extract_ip_net $1)
net2=$(extract_ip_net $2)
[[ $net1 = $net2 ]]
}
_ip=""
_helo=""
_from=""
_to=""
is_cluster=0
grep "^GREY" |
tr "|" "\t" |
cut -f2-5 |
sort -k3,4 -k2 -k1 |
while read ip helo from to
do
if [[ $to = $_to && $from = $_from ]] &&
helo_tld_match "$helo" "$_helo" &&
ip_net_match "$ip" "$_ip"
then
is_cluster=1
elif [[ $is_cluster = 1 ]]
then
is_cluster=0
print_ip_net_with_mask $_ip
fi
_ip="$ip"
_helo="$helo"
_from="$from"
_to="$to"
done
=== spamd-dnsbl ===
#!/bin/sh
#
# Query DNSBL using the IPs in spamdb(8). If an IP is on a black list, add it
# as a TRAPPED entry in the spamdb.
#
# It seems most spammers send once and go away. The 1 minute pass time is
# effective at stopping most of these spammers. The other spammers seem to
# resend 10 minutes to more than an hour later, so a longer pass time won't
# defend against such spammers. That is where DNSBLs can be used to get these
# spammers marked as TRAPPED.
#
# For a list of DNSBL providers, see ~/src/shell/dnsbl-check.
#
# The [bgp-spamd](http://bgp-spamd.net/) project is another option for
# obtaining white and black lists.
#
# TODO: Query multiple DNSBL services and use the results, in agregate, as the
# factor to determine whether the IP should be TRAPPED. For example, if an IP
# is listed in 3 of 8 black lists, then trap it. Maybe we should have levels.
# First query reputable black lists, then fall back to the aggreate. This
# should speed up the process (i.e. no need to query the entire list of
# services if it is listed with a reputable service). The zen.spamhaus.org
# DNSBL seems reputable.
#
START_TIME=~/.${0##*/}.start
start_time=`date +%s`
prev_start_time=$(cat $START_TIME 2>/dev/null || echo 0)
DNSBL="zen.spamhaus.org"
IFS=\|
spamdb | egrep '^(GREY|WHITE)' |
while read entry
do set -- $entry
ip=$2
if [ $1 = GREY ]
then ts=$6
else ts=$5 # WHITE timestamp
fi
if [ $ts -ge $prev_start_time ]
then query=$(IFS="."
set -- $ip
rev_ip="$4.$3.$2.$1"
echo "${rev_ip}.${DNSBL}"
)
host -t A $query >/dev/null && spamdb -ta $ip
#host -t A $query >/dev/null && echo $ip # FOR TESTING
fi
done
echo $start_time > $START_TIME
=== dnsbl-check ===
#!/bin/sh
#
# Check if the given IPv4 address is on a DNS blacklist.
# The list of DNSBL services was taken from
# https://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists.
#
# DNSBLs that return too many false positives:
# - hostkarma.junkemailfilter.com
# - recent.spam.dnsbl.sorbs.net
# - dnsbl.sorbs.net
ip=$1
[[ $ip = [0-9]*.[0-9]*.[0-9]*.[0-9]* ]] || { echo 'IPv4 required'; exit 1; }
rev_ip=$(
IFS="."
set -- $ip
echo "$4.$3.$2.$1"
)
DNSBL_SERVICES='
zen.spamhaus.org
bl.spamcop.net
b.barracudacentral.org
rbl.megarbl.net
all.s5h.net
srnblack.surgate.net
bl.blocklist.de
dnsbl.inps.de
ix.dnsbl.manitu.net
blacklist.hostkarma.com
spamtrap.drbl.drand.net
bl.spamcannibal.org
spam.spamrats.com
dyna.spamrats.com
noptr.spamrats.com
dnsrbl.org
dnsbl.cobion.com
dul.dnsbl.sorbs.net
noservers.dnsbl.sorbs.net
badconf.rhsbl.sorbs.net
escalations.dnsbl.sorbs.net
web.dnsbl.sorbs.net
safe.dnsbl.sorbs.net
babl.rbl.webiron.net
'
for dnsbl in $DNSBL_SERVICES
do host -t A ${rev_ip}.${dnsbl} >/dev/null &&
echo "$ip on $dnsbl black list." &
done
wait
=== admin's crontab ===
*/5 * * * * ~/bin/spamd-dnsbl
0 0 * * * ~/bin/spamdb-stats >> ~/spamdb.stats
0 0 * * * spamdb > ~/spamdb.ark/$(date
+\%m\%d)
*/15 * * * * spamdb | ~/bin/spamclusterd |
doas pfctl -t spamd-cluster -T add -f - -q
=== /etc/doas.conf ===
permit nopass admin cmd pfctl args -t spamd-cluster -T add -f - -q
permit nopass admin cmd pfctl args -t spamd-cluster -T show
permit nopass admin cmd pfctl args -t spamd-cluster -vT show
=== spamdb-stats ===
#!/bin/sh
# Daily spamdb(8) stats via cron.
for i in WHITE GREY TRAP
do spamdb | grep ^$i | wc -l
done | tr "\n" " "
date +"%t%m/%d"
=== spamdb.stats ===
WHITE GREY TRAPPED DATE
82 119 573 11/05
89 50 233 11/06
97 73 172 11/07
101 240 440 11/08
112 161 343 11/09
=== /etc/pf.conf (spamd stuff only) ===
pass out on egress proto tcp to port smtp
pass in on egress proto tcp to port smtp divert-to LOCALHOST port
spamd
pass in log on egress proto tcp from <spamd-white> to port smtp
rdr-to MAIL
pass in log on egress proto tcp from <spamd-cluster> to port smtp
rdr-to MAIL
