Thank you for being patient while I try some different things to find the middle ground.
— Sent from my iPhone > On Sep 5, 2021, at 12:16, clamav.mbou...@spamgourmet.com wrote: > > No problem; good to know it was useful. > > In my case, only the one host behind the NAT (physical PC on a home broadband > connection) is running freshclam anyway, but it appears I was still being > blocked by the rate-limiting. As I understand it, that shouldn't usually > have happened even with the per-IP system. Not sure if that's an issue with > how the new system differentiates between hosts, or perhaps when the download > failed (for whatever reason) freshclam was trying several times and getting > blocked. > > I'm running Linux Mint 20, which is based on Ubuntu 20.04 and uses a lot of > packages from the Ubuntu repositories (upgraded not long after my posts here > a few months ago when I had problems with the default receive timeout in > Ubuntu 16/18.04's packages). ClamAV and freshclam are installed from the > Ubuntu 20.04 repositories, and I haven't yet needed to change the > configuration from the default - so my config will be the same as anyone else > who's installed from the Ubuntu 20.04 repo will have by default. Not sure > whether the new system would have treated everyone with this default config > as the same host, though I'd have thought IP would still be taken into > account as well. > > I'm not complaining - you've clearly had a lot of problems with the CDN being > abused (intentionally or otherwise) and need to try these things. Just trying > to give you whatever information might be useful :) > > Thanks, > Mark. > > > Joel Esler jesler via clamav-users - clamav-users@lists.clamav.net wrote: >> This is useful. Thank you. >> Each host should have a different rate limit under the new system (I turned >> it back off last night, which is why everyone got everything). >> Right now, the rate limit is “per IP”. So, if you have several >> Hosts behind a NAT, so you’ll get blocked. The new system, you can have as >> many hosts behind the same NAT as long as they aren’t using the same config >> file. >> A new problem being, I am seeing a ton of hosts on Amazon or Microsoft’s >> azure that are using the same config, so that’s a new hurdle that those >> people will have to overcome. I am sure there are new problems that we’ll >> encounter during this transition. >> — >> Sent from my iPhone >>>> On Sep 5, 2021, at 09:09, clamav.mbou...@spamgourmet.com wrote: >>> >>> Joel Esler clamav-users@lists.clamav.net wrote: >>>> We are experimenting with a feature that we’ve been working with >>>> Cloudflare on, trying to isolate violators on a per host basis for the >>>> newest versions of ClamAV, instead of IP. >>> >>> I'm guessing you probably already have all the info you need but, in case >>> it happens to be any help, this is what I have in my freshclam logs (on a >>> home desktop PC, so it's not running 24-7)... >>> >>> Last messages from Friday: >>>> Fri Sep 3 22:13:18 2021 -> Received signal: wake up >>>> Fri Sep 3 22:13:18 2021 -> ClamAV update process started at Fri Sep 3 >>>> 22:13:18 2021 >>>> Fri Sep 3 22:13:18 2021 -> WARNING: Your ClamAV installation is OUTDATED! >>>> Fri Sep 3 22:13:18 2021 -> WARNING: Local version: 0.103.2 Recommended >>>> version: 0.103.3 >>>> Fri Sep 3 22:13:18 2021 -> DON'T PANIC! Read >>>> https://www.clamav.net/documents/upgrading-clamav >>>> Fri Sep 3 22:13:18 2021 -> daily.cld database is up-to-date (version: >>>> 26283, sigs: 1970262, f-level: 90, builder: ray >>>> nman) >>>> Fri Sep 3 22:13:18 2021 -> main.cvd database is up-to-date (version: 61, >>>> sigs: 6607162, f-level: 90, builder: sigmgr) >>>> Fri Sep 3 22:13:18 2021 -> bytecode.cvd database is up-to-date (version: >>>> 333, sigs: 92, f-level: 63, builder: awillia >>>> 2) >>>> Fri Sep 3 22:13:18 2021 -> -------------------------------------- >>>> Fri Sep 3 23:06:44 2021 -> Update process terminated >>> >>> So all was up-to-date then. Version 0.103.2 is the latest in the Ubuntu >>> 20.04 repositories, which is why I'm on that version, hence the warning. >>> >>> First messages from Saturday: >>>> Sat Sep 4 11:54:21 2021 -> -------------------------------------- >>>> Sat Sep 4 11:54:21 2021 -> freshclam daemon 0.103.2 (OS: linux-gnu, ARCH: >>>> x86_64, CPU: x86_64) >>>> Sat Sep 4 11:54:21 2021 -> ClamAV update process started at Sat Sep 4 >>>> 11:54:21 2021 >>>> Sat Sep 4 11:54:21 2021 -> WARNING: Your ClamAV installation is OUTDATED! >>>> Sat Sep 4 11:54:21 2021 -> WARNING: Local version: 0.103.2 Recommended >>>> version: 0.103.3 >>>> Sat Sep 4 11:54:21 2021 -> DON'T PANIC! Read >>>> https://www.clamav.net/documents/upgrading-clamav >>>> Sat Sep 4 11:54:21 2021 -> daily database available for update (local >>>> version: 26283, remote version: 26284) >>>> Sat Sep 4 11:54:23 2021 -> WARNING: downloadPatch: Can't download >>>> daily-26284.cdiff from https://database.clamav.net/daily-26284.cdiff >>>> Sat Sep 4 11:54:23 2021 -> The database server doesn't have the latest >>>> patch for the daily database (version 26284). The server will likely have >>>> updated if you check again in a few hours. >>>> Sat Sep 4 11:54:23 2021 -> main.cvd database is up-to-date (version: 61, >>>> sigs: 6607162, f-level: 90, builder: sigmgr) >>>> Sat Sep 4 11:54:23 2021 -> bytecode.cvd database is up-to-date (version: >>>> 333, sigs: 92, f-level: 63, builder: awillia2) >>>> Sat Sep 4 11:54:23 2021 -> -------------------------------------- >>>> Sat Sep 4 12:54:23 2021 -> Received signal: wake up >>>> Sat Sep 4 12:54:23 2021 -> ClamAV update process started at Sat Sep 4 >>>> 12:54:23 2021 >>>> Sat Sep 4 12:54:23 2021 -> WARNING: Your ClamAV installation is OUTDATED! >>>> Sat Sep 4 12:54:23 2021 -> WARNING: Local version: 0.103.2 Recommended >>>> version: 0.103.3 >>>> Sat Sep 4 12:54:23 2021 -> DON'T PANIC! Read >>>> https://www.clamav.net/documents/upgrading-clamav >>>> Sat Sep 4 12:54:23 2021 -> WARNING: FreshClam previously received error >>>> code 429 from the ClamAV Content Delivery Network (CDN). >>>> Sat Sep 4 12:54:23 2021 -> This means that you have been rate limited by >>>> the CDN. >>>> Sat Sep 4 12:54:23 2021 -> 1. Run FreshClam no more than once an hour to >>>> check for updates. >>>> Sat Sep 4 12:54:23 2021 -> FreshClam should check DNS first to see if >>>> an update is needed. >>>> Sat Sep 4 12:54:23 2021 -> 2. If you have more than 10 hosts on your >>>> network attempting to download, >>>> Sat Sep 4 12:54:23 2021 -> it is recommended that you set up a >>>> private mirror on your network using >>>> Sat Sep 4 12:54:23 2021 -> cvdupdate >>>> (https://pypi.org/project/cvdupdate/) to save bandwidth on the >>>> Sat Sep 4 12:54:23 2021 -> CDN and your own network. >>>> Sat Sep 4 12:54:23 2021 -> 3. Please do not open a ticket asking for an >>>> exemption from the rate limit, >>>> Sat Sep 4 12:54:23 2021 -> it will not be granted. >>>> Sat Sep 4 12:54:23 2021 -> WARNING: You are still on cool-down until >>>> after: 2021-09-04 15:54:23 >>> >>> So at 11:54 it determined that an update was available but it couldn't be >>> downloaded. It next checked an hour later at 12:54, and was apparently >>> already rate-limited by then (for 2 checks an hour apart, after none for 12 >>> hours). That was repeated at 13:43 and 14:54, then at 15:54: >>>> Sat Sep 4 15:54:23 2021 -> Received signal: wake up >>>> Sat Sep 4 15:54:23 2021 -> ClamAV update process started at Sat Sep 4 >>>> 15:54:23 2021 >>>> Sat Sep 4 15:54:23 2021 -> WARNING: Your ClamAV installation is OUTDATED! >>>> Sat Sep 4 15:54:23 2021 -> WARNING: Local version: 0.103.2 Recommended >>>> version: 0.103.3 >>>> Sat Sep 4 15:54:23 2021 -> DON'T PANIC! Read >>>> https://www.clamav.net/documents/upgrading-clamav >>>> Sat Sep 4 15:54:23 2021 -> WARNING: Cool-down expired, ok to try again. >>>> Sat Sep 4 15:54:23 2021 -> ERROR: Can't create mirrors.dat in >>>> /var/lib/clamav >>>> Sat Sep 4 15:54:23 2021 -> Hint: The database directory must be writable >>>> for UID XXX or GID YYY >>>> Sat Sep 4 15:54:23 2021 -> daily database available for update (local >>>> version: 26283, remote version: 26284) >>>> Sat Sep 4 15:54:24 2021 -> WARNING: downloadPatch: Can't download >>>> daily-26284.cdiff from https://database.clamav.net/daily-26284.cdiff >>>> Sat Sep 4 15:54:24 2021 -> The database server doesn't have the latest >>>> patch for the daily database (version 26284). The server will likely have >>>> updated if you check again in a few hours. >>>> Sat Sep 4 15:54:24 2021 -> main.cvd database is up-to-date (version: 61, >>>> sigs: 6607162, f-level: 90, builder: sigmgr) >>>> Sat Sep 4 15:54:24 2021 -> bytecode.cvd database is up-to-date (version: >>>> 333, sigs: 92, f-level: 63, builder: awillia2) >>>> Sat Sep 4 15:54:24 2021 -> -------------------------------------- >>> >>> At 16:54, 17:54 and 18:54 it was back to "FreshClam previously received >>> error code 429... you have been rate limited by the CDN". At 19:54 the >>> cool-down expired and it was able to check again - but failed again the >>> same as above. Then on cool-down at 20:54, 21:54 and 22:54, after which >>> the PC was shut down. This is the only instance of freshclam running on my >>> home network, and nothing else should be attempting to download the ClamAV >>> databases (I haven't been trying to download them manually, or running >>> other instances of freshclam). >>> >>> Today: >>>> Sun Sep 5 11:27:13 2021 -> -------------------------------------- >>>> Sun Sep 5 11:27:13 2021 -> freshclam daemon 0.103.2 (OS: linux-gnu, ARCH: >>>> x86_64, CPU: x86_64) >>>> Sun Sep 5 11:27:13 2021 -> ClamAV update process started at Sun Sep 5 >>>> 11:27:13 2021 >>>> Sun Sep 5 11:27:13 2021 -> WARNING: Your ClamAV installation is OUTDATED! >>>> Sun Sep 5 11:27:13 2021 -> WARNING: Local version: 0.103.2 Recommended >>>> version: 0.103.3 >>>> Sun Sep 5 11:27:13 2021 -> DON'T PANIC! Read >>>> https://www.clamav.net/documents/upgrading-clamav >>>> Sun Sep 5 11:27:13 2021 -> daily database available for update (local >>>> version: 26283, remote version: 26285) >>>> Sun Sep 5 11:27:15 2021 -> Testing database: >>>> '/var/lib/clamav/tmp.a9599a4ff7/clamav-431aa03fce17054479c616a2f44eae7b.tmp-daily.cld' >>>> ... >>>> Sun Sep 5 11:27:20 2021 -> Database test passed. >>>> Sun Sep 5 11:27:22 2021 -> daily.cld updated (version: 26285, sigs: >>>> 1970840, f-level: 90, builder: raynman) >>>> Sun Sep 5 11:27:22 2021 -> main.cvd database is up-to-date (version: 61, >>>> sigs: 6607162, f-level: 90, builder: sigmgr) >>>> Sun Sep 5 11:27:22 2021 -> bytecode.cvd database is up-to-date (version: >>>> 333, sigs: 92, f-level: 63, builder: awillia2) >>>> Sun Sep 5 11:27:22 2021 -> WARNING: Clamd was NOT notified: Can't connect >>>> to clamd through /var/run/clamav/clamd.ctl: No such file or directory >>>> Sun Sep 5 11:27:22 2021 -> -------------------------------------- >>>> Sun Sep 5 12:27:23 2021 -> Received signal: wake up >>>> Sun Sep 5 12:27:23 2021 -> ClamAV update process started at Sun Sep 5 >>>> 12:27:23 2021 >>>> Sun Sep 5 12:27:23 2021 -> WARNING: Your ClamAV installation is OUTDATED! >>>> Sun Sep 5 12:27:23 2021 -> WARNING: Local version: 0.103.2 Recommended >>>> version: 0.103.3 >>>> Sun Sep 5 12:27:23 2021 -> DON'T PANIC! Read >>>> https://www.clamav.net/documents/upgrading-clamav >>>> Sun Sep 5 12:27:23 2021 -> daily.cld database is up-to-date (version: >>>> 26285, sigs: 1970840, f-level: 90, builder: raynman) >>>> Sun Sep 5 12:27:23 2021 -> main.cvd database is up-to-date (version: 61, >>>> sigs: 6607162, f-level: 90, builder: sigmgr) >>>> Sun Sep 5 12:27:23 2021 -> bytecode.cvd database is up-to-date (version: >>>> 333, sigs: 92, f-level: 63, builder: awillia2) >>>> Sun Sep 5 12:27:23 2021 -> -------------------------------------- >>> >>> So it was able to successfully update today. >>> >>> -- >>> Mark. >>> >>> >>> _______________________________________________ >>> >>> clamav-users mailing list >>> clamav-users@lists.clamav.net >>> https://lists.clamav.net/mailman/listinfo/clamav-users >>> >>> >>> Help us build a comprehensive ClamAV guide: >>> https://github.com/vrtadmin/clamav-faq >>> >>> http://www.clamav.net/contact.html#ml >> _______________________________________________ >> clamav-users mailing list >> clamav-users@lists.clamav.net >> https://lists.clamav.net/mailman/listinfo/clamav-users >> Help us build a comprehensive ClamAV guide: >> https://github.com/vrtadmin/clamav-faq >> http://www.clamav.net/contact.html#ml > > > _______________________________________________ > > clamav-users mailing list > clamav-users@lists.clamav.net > https://lists.clamav.net/mailman/listinfo/clamav-users > > > Help us build a comprehensive ClamAV guide: > https://github.com/vrtadmin/clamav-faq > > http://www.clamav.net/contact.html#ml _______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
