On 20.09.18 16:05, Matus UHLAR - fantomas wrote:
I looked at update times and they are different each day - debian script
sleeps random number of seconds (up to one hour) in order to lower the
impact at mirror servers.
I have removed the "--fail" option from curl and will look at error message
if there's any.
I'll keep you updated and will fill bugreport if I'm able to find out
anything useful.
I was able to repeat this problem now:
# /usr/bin/curl --verbose -L -O --remote-time -g --max-redirs 2
--connect-timeout 30 --max-time 300 -o 1843052.tar.gz --
http://sa-update.spamassassin.org/1843052.tar.gz
* Hostname was NOT found in DNS cache
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0*
Trying 64.142.56.146...
* Connected to sa-update.spamassassin.org (64.142.56.146) port 80 (#0)
GET /1843052.tar.gz HTTP/1.1
User-Agent: curl/7.38.0
Host: sa-update.spamassassin.org
Accept: */*
0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0<
HTTP/1.1 200 OK
< Date: Mon, 08 Oct 2018 14:16:19 GMT
* Server Apache/2.4.6 (CentOS) is not blacklisted
< Server: Apache/2.4.6 (CentOS)
< Last-Modified: Mon, 08 Oct 2018 03:19:20 GMT
< ETag: "4600c-577af16429e00"
< Accept-Ranges: bytes
< Content-Length: 286732
< Content-Type: application/x-gzip
<
{ [data not shown]
0 280k 0 1 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0*
transfer closed with 286731 bytes remaining to read
* Closing connection 0
curl: (18) transfer closed with 286731 bytes remaining to read
# ls -l 1843052.tar.gz
-rw-r--r-- 1 root root 1 Oct 8 16:16 1843052.tar.gz
look at today's debug log says:
Oct 8 07:12:59.899 [20257] dbg: channel: selected mirror
http://sa-update.spamassassin.org
Oct 8 07:12:59.899 [20257] dbg: http: url:
http://sa-update.spamassassin.org/1843052.tar.gz
Oct 8 07:12:59.899 [20257] dbg: http: downloading to:
/var/lib/spamassassin/3.004000/updates_spamassassin_org/1843052.tar.gz, new
Oct 8 07:12:59.899 [20257] dbg: util: executable for curl was found at
/usr/bin/curl
Oct 8 07:12:59.899 [20257] dbg: http: /usr/bin/curl -s -L -O --remote-time -g
--max-redirs 2 --connect-timeout 30 --max-time 300 -o 1843052.tar.gz --
http://sa-update.spamassassin.org/1843052.tar.gz
Oct 8 07:13:15.385 [20257] dbg: http: process [20258], exit status: 4608
Oct 8 07:13:15.385 [20257] dbg: channel: selected mirror
http://sa-update.ena.com
Oct 8 07:13:15.385 [20257] dbg: http: url:
http://sa-update.ena.com/1843052.tar.gz
Oct 8 07:13:15.385 [20257] dbg: http: downloading to:
/var/lib/spamassassin/3.004000/updates_spamassassin_org/1843052.tar.gz, update
Oct 8 07:13:15.385 [20257] dbg: util: executable for curl was found at
/usr/bin/curl
Oct 8 07:13:15.385 [20257] dbg: http: /usr/bin/curl -s -L -O --remote-time -g
--max-redirs 2 --connect-timeout 30 --max-time 300 -o 1843052.tar.gz -z
1843052.tar.gz -- http://sa-update.ena.com/1843052.tar.gz
Oct 8 07:13:15.889 [20257] dbg: http: process [20272], exit status: 0
This looks that invalid file was downloaded from sa-update.spamassassin.org,
and while next curl invocation succeeded with exit code 0, the file was not
overridden:
# /usr/bin/curl -s -L -O --remote-time -g --max-redirs 2 --connect-timeout 30
--max-time 300 -o 1843052.tar.gz -z 1843052.tar.gz --
http://sa-update.ena.com/1843052.tar.gz
# ls -l 1843052.tar.gz
-rw-r--r-- 1 root root 243 Oct 8 16:21 1843052.tar.gz
# /usr/bin/curl -s -L -O --remote-time -g --max-redirs 2 --connect-timeout 30
--max-time 300 -o 1843052.tar.gz -z 1843052.tar.gz --
http://sa-update.ena.com/1843052.tar.gz
# ls -l 1843052.tar.gz
-rw-r--r-- 1 root root 243 Oct 8 16:21 1843052.tar.gz
# rm 1843052.tar.gz
# /usr/bin/curl -s -L -O --remote-time -g --max-redirs 2 --connect-timeout 30
--max-time 300 -o 1843052.tar.gz -z 1843052.tar.gz --
http://sa-update.ena.com/1843052.tar.gz
# ls -l 1843052.tar.gz
-rw-r--r-- 1 root root 286732 Oct 8 05:19 1843052.tar.gz
(the file size changed to 243 because of my tests).
further look at logs says that all failed downloads were from
sa-update.spamassassin.org:
Sep 28 07:43:07.888 [7018] dbg: http: /usr/bin/curl -s -L -O --remote-time -g
--max-redirs 2 --connect-timeout 30 --max-time 300 -o 1842077.tar.gz --
http://sa-update.spamassassin.org/1842077.tar.gz
Sep 28 07:43:21.973 [7018] dbg: http: process [7019], exit status: 4608
Oct 5 06:35:10.552 [29702] dbg: http: /usr/bin/curl -s -L -O --remote-time -g
--max-redirs 2 --connect-timeout 30 --max-time 300 -o 1842787.tar.gz --
http://sa-update.spamassassin.org/1842787.tar.gz
Oct 5 06:35:29.199 [29702] dbg: http: process [29705], exit status: 4608
Oct 7 07:17:37.644 [30424] dbg: http: /usr/bin/curl -s -L -O --remote-time -g
--max-redirs 2 --connect-timeout 30 --max-time 300 -o 1843008.tar.gz --
http://sa-update.spamassassin.org/1843008.tar.gz
Oct 7 07:18:11.394 [30424] dbg: http: process [30427], exit status: 4608
btw, when repeating attempts to download from sa-update.spamassassin.org,
many of them failed.
It looks to me that:
1. sa-update.spamassassin.org has some rate limiting or problem with tcp/ip
(accelerator?)
2. the script does NOT remove downloaded file when download fails and curl
is NOT instructed to do so.
--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
One OS to rule them all, One OS to find them,
One OS to bring them all and into darkness bind them
