Mark, Thanks for tracking down the freshclam.conf issue and submitting the bug report.
With regards to the 5 second between attempts, I'm not sure either. It would seem reasonable if there was some sort of network glitch, but if it's a persistent issue like the receive timeout for slower connections, then yeah 5 seconds doesn't make much sense. I'm not really sure what to say. We could reduce the number or retries attempts as well, but in the end the config change to "ReceiveTimeout=0" should resolve the issue and no one should have to retry. I'm inclined to leave freshclam as-is. Regarding the mirrors.dat ownership issue: You're probably right. It probably tries to update mirrors.dat later on after it has switched to run as "clamav" and then fails. :-( Well, we'll get this fixed in the next patch release. Sorry about the trouble. -Micah > -----Original Message----- > From: clamav-users <clamav-users-boun...@lists.clamav.net> On Behalf Of > clamav.mbou...@spamgourmet.com > Sent: Friday, May 7, 2021 1:33 PM > To: clamav-users@lists.clamav.net > Subject: Re: [clamav-users] Update on rate limits and downloading > > Hi Micah, > > Thanks for the info. It looks like the timeout is an Ubuntu packaging issue. > The > post-install scripts for the Ubuntu 16.04 and 18.04 clamav-freshclam 0.103.2 > packages create a freshclam.conf with "ReceiveTimeout=30", while the Ubuntu > 20.04 package sets "ReceiveTimeout=0". I hadn't thought there would be a > difference between the packages for different versions of Ubuntu, since > they're > all ClamAV/FreshClam version 0.103.2. I've raised a bug on launchpad > <https://bugs.launchpad.net/ubuntu/+source/clamav/+bug/1927777> to > suggest at least updating the 18.04 package (and the 16.04 one if it's still > getting updates now that 16.04 itself is end-of-life). > > I'm still not sure whether FreshClam should be holding off for a bit longer > than > 5 seconds between attempts, to avoid triggering the rate limiting? The > closest > thing I can see in the configuration is "MaxAttempts 5". Although from the > log, > it looks like it tries 5 times with 5 seconds between attempts, says "Giving > up > on https://database.clamav.net...";, but then immediately starts trying again > and triggers the rate limiting after a few more attempts. > > I haven't run freshclam manually at all, it's only ever been run by the > clamav- > freshclam systemd service installed by the package. It doesn't look like the > service configuration specifies a user, so it's presumably starting as root, > but > freshclam is then dropping privileges to the "clamav" user after starting > (freshclam.conf includes "DatabaseOwner clamav"). It looks like it might be > creating the file as root before dropping privileges, and then trying to > update it > later - probably hitting the same condition as you mention when running it via > sudo and then in other ways. > > Thanks, > Mark. > > > Micah Snyder micasnyd via clamav-users wrote: > > Hi Mark, > > > > I'm not sure how you got a config with the default set to "ReceiveTimeout > 30". I just tested with ubunte 20.04 a moment ago and a fresh `apt install > clamav` (0.103.2). The config I found in /etc/clamav/freshclam.conf has > "ReceiveTimeout 0" which means it is disabled. ClamAV's built-in default (if > you > don't specify) is also "0". So I'm not really sure what went wrong for you. > > > > So we do have a minor problem with the mirrors.dat in 0.103.2. It will be > owned by root instead of by the "clamav" user if you run "sudo freshclam -- > daemon". Then if you try running freshclam a different way, you may run into > permissions issues. We'll have to fix this in the next patch version ☹. > > > > -Micah > > > >> -----Original Message----- > >> From: clamav-users <clamav-users-boun...@lists.clamav.net> On Behalf Of > >> clamav.mbou...@spamgourmet.com > >> Sent: Thursday, May 6, 2021 1:12 PM > >> To: clamav-users@lists.clamav.net > >> Subject: Re: [clamav-users] Update on rate limits and downloading > >> > >> Joel Esler jesler via clamav-users wrote: > >>> Overall — we’re doing much better. > >>> > >>> We’ve reduced the amount of bandwidth we’re serving by 4x, so we’ve > >>> made significant progress. > >>> > >>> /However, /we still have over 700 individual systems downloading the > >>> full daily.cvd over 200x a day. (This should be once a day, /if > >>> that/.) > >>> > >>> If you are not using 0.103.2 and it’s accompanying FreshClam to > >>> download these updates, and when you do create a NEW FreshClam.conf > >>> file and move your settings to that. We’re going to have to start > >>> blocking these atrocious abusers, as the rate limits are hurting > >>> everyone else at this point. > >> > >> I'm new to installing ClamAV, so there may be something I haven't done > >> quite right here. A couple of weeks ago, I installed ClamAV 0.103.2 from > >> the > >> Ubuntu repositories (clamav, clamav-freshclam, clamav-daemon, clamav- > >> docs, clamtk and libclamunrar9 packages). > >> > >> By default, FreshClam seems to use too short a download timeout and retry > >> too frequently, triggering the rate limiting. After installing, the > >> FreshClam > >> service would repeatedly attempt to download the daily.cvd file, time out > >> after 30 seconds, and wait 5 seconds before trying again. > >> After a few attempts, it then gets blocked by the CDN (if that's what "you > are > >> on cool-down" in the log means?) for 4 hours. By the time I'd realised > >> this > >> was happening following the initial install, I was already blocked. > >> > >> Perhaps this might, if left in a default configuration, be seen to attempt > >> to > >> download daily.cvd over 100 times a day, but without ever actually getting > >> the whole file. From what I'd seen here and in documentation / FAQs, I > >> thought FreshClam was supposed to avoid retrying so frequently that it > >> triggers the rate limiting? > >> > >> I don't know if the default configuration is provided by ClamAV or the > >> Ubuntu packaging (either way, it seems FreshClam shouldn't just keep > >> retrying so quickly?) In my case, freshclam.conf originally had > >> "ReceiveTimeout 30". Increasing it to 60 wasn't enough. I then went to > 600, > >> which was successful. Somewhere in between would probably have been > >> fine, but incrementing more gradually would have been a long process, > >> having to wait at least 4 hours between attempts (particularly as > >> restarting > >> FreshClam after setting a new timeout seems to get blocked for a further 4 > >> hours - not just the remainder of the original block). > >> > >> In case it's of any use (and if this list allows it), I've attached my > freshclam.log > >> from those initial attempts. > >> > >> All seems to be working OK now, but posting here in case the information is > >> useful. > >> > >>> Please help us, stay diligent, keep going keep upgrading. Upgrade to > >>> 0.103.2, and keep your mirrors.dat file around, this file contains a > >>> snapshot of where you are in your update progression so that the next > >>> time that FreshClam run, it can start where it left off. > >> > >> Interesting you should mention mirrors.dat... Aside from the downloads > >> timing out, there are also some errors in my freshclam.log about not being > >> able to create mirrors.dat. That's a bit odd, since the /var/lib/clamav/ > >> directory is owned and writeable by the correct user, but the mirrors.dat > >> file > >> within it is owned by root. Deleting that file and restarting the > >> freshclam > >> service, the mirrors.dat file gets recreated, again owned by root. That > >> error > >> hasn't appeared in the logs since, although mirrors.dat is still dated 25th > April, > >> so I'm not sure if there's still a problem with that. > >> > >> -- > >> Mark. > > > > _______________________________________________ > > > > clamav-users mailing list > > clamav-users@lists.clamav.net > > https://lists.clamav.net/mailman/listinfo/clamav-users > > > > > > Help us build a comprehensive ClamAV guide: > > https://github.com/vrtadmin/clamav-faq > > > > http://www.clamav.net/contact.html#ml > > > > > _______________________________________________ > > clamav-users mailing list > clamav-users@lists.clamav.net > https://lists.clamav.net/mailman/listinfo/clamav-users > > > Help us build a comprehensive ClamAV guide: > https://github.com/vrtadmin/clamav-faq > > http://www.clamav.net/contact.html#ml _______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
