I have changed the way we use freshclam to mitigate the sync problem
with the new Cloudflare mirror regime -- which, by the way, *still*
seems to lag what the DNS TXT record reports.
What I have done is to introduce a pretesting phase before invoking
freshclam. Our new update method operates in three steps:
First it queries the DNS TXT record and compares those version numbers
with the local cvd version numbers.
Second, if any of the DNS TXT version numbers is greater the the local
one, it uses curl to retrieve the version number from the corresponding
cvd file at database.clamav.net. (A variation on what Dennis Peterson
suggested.)
Third, if -- and only if -- the version number in that actual cvd file
is greater than the version number in the local cvd, it finally invokes
freshclam. (Unfortunately, since there a several Cloudflare servers,
there is no way to guarantee the server freshclam uses is as up to date
as the one the curl used.)
As can be seen in the log excerpt below, the DNS TXT record is again
premature in claiming that there is a new cvd available.
The log lines are laid out as follows. LCLver is the local version,
EXTver is the version on the server (as reported by curl), the keywords
DNS, EXT just mean who reported the versions, and UPD means update via
freshclam.
--> DNS Daily DNSver/LCLver Bytecode DNSver/LCLver Main DNSver/LCLver
--> EXT Daily EXTver/DNSver/LCLver Bytecode EXTver/DNSver/LCLver Main
EXTver/DNSver/LCLver
--> UPD Daily EXTver/DNSver/LCLver Bytecode EXTver/DNSver/LCLver Main
EXTver/DNSver/LCLver
The (added by me) comment lines (#) below show that the DNS and
Cloudflare server are out of sync again.
=======================================================================
------------------------------ Monday 09 July 2018 at 08:48:01
------------------------------
/opt/clamav/bin/testclam-external
--> DNS D 24736/24736 B 322/322 M 58/58
------------------------------ Monday 09 July 2018 at 09:03:01
------------------------------
/opt/clamav/bin/testclam-external
--> EXT D 24736/24737/24736 B 322/322/322 M 58/58/58
# ^^^^^ ^^^^^
# curl DNS
------------------------------ Monday 09 July 2018 at 09:18:01
------------------------------
/opt/clamav/bin/testclam-external
--> UPD D 24737/24737/24736 B 322/322/322 M 58/58/58
removed `/opt/clamav/share/clamav/mirrors.dat'
/opt/clamav/bin/freshclam -v --stdout --on-update-execute=EXIT_1
Current working dir is /opt/clamav.d/clamav.0.100.0/share/clamav
Max retries == 1
ClamAV update process started at Mon Jul 9 09:18:03 2018
Using IPv6 aware code
Querying current.cvd.clamav.net
TTL: 898
Software version from DNS: 0.100.0
main.cvd version from DNS: 58
main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder:
sigmgr)
daily.cvd version from DNS: 24737
Retrieving http://database.clamav.net/daily.cvd
Using ip '10.11.14.160' for fetching.
Trying to download http://database.clamav.net/daily.cvd (IP: 104.16.185.138)
Downloading daily.cvd [100%]
Loading signatures from daily.cvd
Properly loaded 2008285 signatures from new daily.cvd
daily.cvd updated (version: 24737, sigs: 2008285, f-level: 63, builder: neo)
Querying daily.24737.91.1.0.6810B98A.ping.clamav.net
bytecode.cvd version from DNS: 322
bytecode.cvd is up to date (version: 322, sigs: 90, f-level: 63, builder: neo)
Database updated (6574624 signatures) from database.clamav.net (IP:
104.16.185.138)
OnUpdateExecute: EXIT_1
------------------------------ Monday 09 July 2018 at 09:18:17
------------------------------
=======================================================================
P.S. Our new method is triggered by cron more frequently than we used
to simply run freshclam in the past, but since the DNS TXT query is
extremely low cost, and freshclam is only run when the quick curl says
the cvd is really new, the total load on the ClamAV server will be less
compared to the many freshclam sync failures previously.
_______________________________________________
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml
