-----Original Message-----
From: clamav-users <clamav-users-boun...@lists.clamav.net> On Behalf Of
clamav.mbou...@spamgourmet.com
Sent: Thursday, May 6, 2021 1:12 PM
To: clamav-users@lists.clamav.net
Subject: Re: [clamav-users] Update on rate limits and downloading
Joel Esler jesler via clamav-users wrote:
Overall — we’re doing much better.
We’ve reduced the amount of bandwidth we’re serving by 4x, so we’ve
made significant progress.
/However, /we still have over 700 individual systems downloading the
full daily.cvd over 200x a day. (This should be once a day, /if
that/.)
If you are not using 0.103.2 and it’s accompanying FreshClam to
download these updates, and when you do create a NEW FreshClam.conf
file and move your settings to that. We’re going to have to start
blocking these atrocious abusers, as the rate limits are hurting
everyone else at this point.
I'm new to installing ClamAV, so there may be something I haven't done
quite right here. A couple of weeks ago, I installed ClamAV 0.103.2 from the
Ubuntu repositories (clamav, clamav-freshclam, clamav-daemon, clamav-
docs, clamtk and libclamunrar9 packages).
By default, FreshClam seems to use too short a download timeout and retry
too frequently, triggering the rate limiting. After installing, the FreshClam
service would repeatedly attempt to download the daily.cvd file, time out
after 30 seconds, and wait 5 seconds before trying again.
After a few attempts, it then gets blocked by the CDN (if that's what "you are
on cool-down" in the log means?) for 4 hours. By the time I'd realised this
was happening following the initial install, I was already blocked.
Perhaps this might, if left in a default configuration, be seen to attempt to
download daily.cvd over 100 times a day, but without ever actually getting
the whole file. From what I'd seen here and in documentation / FAQs, I
thought FreshClam was supposed to avoid retrying so frequently that it
triggers the rate limiting?
I don't know if the default configuration is provided by ClamAV or the
Ubuntu packaging (either way, it seems FreshClam shouldn't just keep
retrying so quickly?) In my case, freshclam.conf originally had
"ReceiveTimeout 30". Increasing it to 60 wasn't enough. I then went to 600,
which was successful. Somewhere in between would probably have been
fine, but incrementing more gradually would have been a long process,
having to wait at least 4 hours between attempts (particularly as restarting
FreshClam after setting a new timeout seems to get blocked for a further 4
hours - not just the remainder of the original block).
In case it's of any use (and if this list allows it), I've attached my
freshclam.log
from those initial attempts.
All seems to be working OK now, but posting here in case the information is
useful.
Please help us, stay diligent, keep going keep upgrading. Upgrade to
0.103.2, and keep your mirrors.dat file around, this file contains a
snapshot of where you are in your update progression so that the next
time that FreshClam run, it can start where it left off.
Interesting you should mention mirrors.dat... Aside from the downloads
timing out, there are also some errors in my freshclam.log about not being
able to create mirrors.dat. That's a bit odd, since the /var/lib/clamav/
directory is owned and writeable by the correct user, but the mirrors.dat file
within it is owned by root. Deleting that file and restarting the freshclam
service, the mirrors.dat file gets recreated, again owned by root. That error
hasn't appeared in the logs since, although mirrors.dat is still dated 25th
April,
so I'm not sure if there's still a problem with that.
--
Mark.
