from the email:
FAILURE DUMP SUMMARY:
coyote /GenesAmandaHelper-0.61/config-bak lev 0 partial taper: source
server crc (6158b8f5:29861110032) and input server crc
(4f171223:29861110032) differ)
coyote /GenesAmandaHelper-0.61/config-bak lev 0 FAILED [data timeout]
coyote /GenesAmandaHelper-0.61/config-bak lev 0 partial taper:
successfully taped a partial dump
coyote /opt lev 0 partial taper: source server crc
(1f484cf4:15602100421) and input server crc (a7bc8db8:15602100421)
differ)
coyote /opt lev 0 FAILED [data timeout]
coyote /opt lev 0 FAILED [failed to set shm-ring]
coyote /usr/movies lev 0 partial taper: source server crc
(55338628:13370009600) and input server crc (feecb660:13370009600)
differ)
coyote /usr/movies lev 0 was successfully retried
And:
FAILED DUMP DETAILS:
/-- coyote /GenesAmandaHelper-0.61/config-bak lev 0 FAILED [data
timeout]
sendbackup: info BACKUP=APPLICATION
sendbackup: info APPLICATION=amgtar
sendbackup: info
RECOVER_CMD=/bin/gzip -dc |/usr/local/libexec/amanda/application/amgtar
restore [./file-to-restore]+
sendbackup: info COMPRESS_SUFFIX=.gz
sendbackup: info end
\--------
/-- coyote /opt lev 0 FAILED [data timeout]
sendbackup: info BACKUP=APPLICATION
sendbackup: info APPLICATION=amgtar
sendbackup: info
RECOVER_CMD=/bin/gzip -dc |/usr/local/libexec/amanda/application/amgtar
restore [./file-to-restore]+
sendbackup: info COMPRESS_SUFFIX=.gz
sendbackup: info end
\--------
NOTES:
planner: Incremental of coyote:/usr/local bumped to level 2.
driver: coyote /GenesAmandaHelper-0.61/config-bak 20210122020104 0
[Will retry dump because of holding disk error: source server crc
(6158b8f5:29861110032) and input server crc (4f171223:29861110032)
differ)]
driver: coyote /opt 20210122020104 0 [Will retry dump because of
holding disk error: source server crc (1f484cf4:15602100421) and input
server crc (a7bc8db8:15602100421) differ)]
driver: coyote /usr/movies 20210122020104 0 [Will retry dump because of
holding disk error: source server crc (55338628:13370009600) and input
server crc (feecb660:13370009600) differ)]
taper: tape Dailys-2 kb 73355394 fm 79 [OK]
big estimate: coyote /GenesAmandaHelper-0.61/config-bak 0
est: 28000M out 0M
left in the holding disk:
root@coyote:data$ ls -l /sdb/dumps/20210122020104/
total 57456372
-rw------- 1 amanda amanda 1048576000 Jan 22 02:31
coyote._GenesAmandaHelper-0.61_config-bak.0
-rw------- 1 amanda amanda 1048576000 Jan 22 02:08
coyote._GenesAmandaHelper-0.61_config-bak.0.1
-rw------- 1 amanda amanda 1048576000 Jan 22 02:16
coyote._GenesAmandaHelper-0.61_config-bak.0.10
-rw------- 1 amanda amanda 1048576000 Jan 22 02:17
coyote._GenesAmandaHelper-0.61_config-bak.0.11
-rw------- 1 amanda amanda 1048576000 Jan 22 02:18
coyote._GenesAmandaHelper-0.61_config-bak.0.12
-rw------- 1 amanda amanda 1048576000 Jan 22 02:19
coyote._GenesAmandaHelper-0.61_config-bak.0.13
-rw------- 1 amanda amanda 1048576000 Jan 22 02:20
coyote._GenesAmandaHelper-0.61_config-bak.0.14
-rw------- 1 amanda amanda 1048576000 Jan 22 02:21
coyote._GenesAmandaHelper-0.61_config-bak.0.15
-rw------- 1 amanda amanda 1048576000 Jan 22 02:22
coyote._GenesAmandaHelper-0.61_config-bak.0.16
-rw------- 1 amanda amanda 1048576000 Jan 22 02:23
coyote._GenesAmandaHelper-0.61_config-bak.0.17
-rw------- 1 amanda amanda 1048576000 Jan 22 02:23
coyote._GenesAmandaHelper-0.61_config-bak.0.18
-rw------- 1 amanda amanda 1048576000 Jan 22 02:24
coyote._GenesAmandaHelper-0.61_config-bak.0.19
-rw------- 1 amanda amanda 1048576000 Jan 22 02:09
coyote._GenesAmandaHelper-0.61_config-bak.0.2
-rw------- 1 amanda amanda 1048576000 Jan 22 02:25
coyote._GenesAmandaHelper-0.61_config-bak.0.20
-rw------- 1 amanda amanda 1048576000 Jan 22 02:26
coyote._GenesAmandaHelper-0.61_config-bak.0.21
-rw------- 1 amanda amanda 1048576000 Jan 22 02:27
coyote._GenesAmandaHelper-0.61_config-bak.0.22
-rw------- 1 amanda amanda 1048576000 Jan 22 02:28
coyote._GenesAmandaHelper-0.61_config-bak.0.23
-rw------- 1 amanda amanda 1048576000 Jan 22 02:28
coyote._GenesAmandaHelper-0.61_config-bak.0.24
-rw------- 1 amanda amanda 1048576000 Jan 22 02:29
coyote._GenesAmandaHelper-0.61_config-bak.0.25
-rw------- 1 amanda amanda 1048576000 Jan 22 02:30
coyote._GenesAmandaHelper-0.61_config-bak.0.26
-rw------- 1 amanda amanda 1048576000 Jan 22 02:31
coyote._GenesAmandaHelper-0.61_config-bak.0.27
-rw------- 1 amanda amanda 501932304 Jan 22 02:31
coyote._GenesAmandaHelper-0.61_config-bak.0.28
-rw------- 1 amanda amanda 1048576000 Jan 22 02:10
coyote._GenesAmandaHelper-0.61_config-bak.0.3
-rw------- 1 amanda amanda 1048576000 Jan 22 02:11
coyote._GenesAmandaHelper-0.61_config-bak.0.4
-rw------- 1 amanda amanda 1048576000 Jan 22 02:12
coyote._GenesAmandaHelper-0.61_config-bak.0.5
-rw------- 1 amanda amanda 1048576000 Jan 22 02:13
coyote._GenesAmandaHelper-0.61_config-bak.0.6
-rw------- 1 amanda amanda 1048576000 Jan 22 02:14
coyote._GenesAmandaHelper-0.61_config-bak.0.7
-rw------- 1 amanda amanda 1048576000 Jan 22 02:15
coyote._GenesAmandaHelper-0.61_config-bak.0.8
-rw------- 1 amanda amanda 1048576000 Jan 22 02:16
coyote._GenesAmandaHelper-0.61_config-bak.0.9
-rw------- 1 amanda amanda 1048576000 Jan 22 02:54 coyote._opt.0
-rw------- 1 amanda amanda 1048576000 Jan 22 02:33 coyote._opt.0.1
-rw------- 1 amanda amanda 1048576000 Jan 22 02:49 coyote._opt.0.10
-rw------- 1 amanda amanda 1048576000 Jan 22 02:49 coyote._opt.0.11
-rw------- 1 amanda amanda 1048576000 Jan 22 02:50 coyote._opt.0.12
-rw------- 1 amanda amanda 1048576000 Jan 22 02:51 coyote._opt.0.13
-rw------- 1 amanda amanda 922527941 Jan 22 02:54 coyote._opt.0.14
-rw------- 1 amanda amanda 1048576000 Jan 22 02:34 coyote._opt.0.2
-rw------- 1 amanda amanda 1048576000 Jan 22 02:35 coyote._opt.0.3
-rw------- 1 amanda amanda 1048576000 Jan 22 02:38 coyote._opt.0.4
-rw------- 1 amanda amanda 1048576000 Jan 22 02:41 coyote._opt.0.5
-rw------- 1 amanda amanda 1048576000 Jan 22 02:43 coyote._opt.0.6
-rw------- 1 amanda amanda 1048576000 Jan 22 02:45 coyote._opt.0.7
-rw------- 1 amanda amanda 1048576000 Jan 22 02:46 coyote._opt.0.8
-rw------- 1 amanda amanda 1048576000 Jan 22 02:47 coyote._opt.0.9
-rw------- 1 amanda amanda 1048576000 Jan 22 03:59 coyote._usr_movies.0
-rw------- 1 amanda amanda 1048576000 Jan 22 03:55 coyote._usr_movies.0.1
-rw------- 1 amanda amanda 1048576000 Jan 22 03:58
coyote._usr_movies.0.10
-rw------- 1 amanda amanda 1048576000 Jan 22 03:58
coyote._usr_movies.0.11
-rw------- 1 amanda amanda 787523584 Jan 22 03:59
coyote._usr_movies.0.12
-rw------- 1 amanda amanda 1048576000 Jan 22 03:55 coyote._usr_movies.0.2
-rw------- 1 amanda amanda 1048576000 Jan 22 03:56 coyote._usr_movies.0.3
-rw------- 1 amanda amanda 1048576000 Jan 22 03:56 coyote._usr_movies.0.4
-rw------- 1 amanda amanda 1048576000 Jan 22 03:56 coyote._usr_movies.0.5
-rw------- 1 amanda amanda 1048576000 Jan 22 03:56 coyote._usr_movies.0.6
-rw------- 1 amanda amanda 1048576000 Jan 22 03:56 coyote._usr_movies.0.7
-rw------- 1 amanda amanda 1048576000 Jan 22 03:56 coyote._usr_movies.0.8
-rw------- 1 amanda amanda 1048576000 Jan 22 03:57 coyote._usr_movies.0.9
root@coyote:data$
And I was asked to show the amstatus output when it failed:
root@coyote:data$ cat /home/amanda/log/amstat.d/amstat-210122-0507
Using: /usr/local/var/amanda/Daily/amdump.1
Thats it, normally its about 10k of stuff
One of the failed ones header:
root@coyote:data$ dd if=00046.coyote._opt.0 bs=32k count=1
AMANDA: SPLIT_FILE 20210122020104 coyote /opt part 1/-1 lev 0 comp .gz
program APPLICATION
APPLICATION=amgtar
ORIGSIZE=24681320
NATIVE-CRC=b5751dc0:25273671680
CLIENT-CRC=a7bc8db8:15602100421
SERVER-CRC=a7bc8db8:15602100421
DLE=<<ENDDLE
<dle>
<program>APPLICATION</program>
<disk>/opt</disk>
<level>0</level>
<auth>bsdtcp</auth>
<compress>BEST</compress>
<record>YES</record>
<index>YES</index>
<datapath>AMANDA</datapath>
<exclude>
<list>/GenesAmandaHelper-0.61/excludes</list>
</exclude>
<backup-program>
<plugin>amgtar</plugin>
<property>
<name>ignore</name>
<value encoding="raw"
raw="OiBzb2NrZXQgaWdub3JlZCQ=">:_socket_ignored$</value> <value
encoding="raw"
raw="ZmlsZSBjaGFuZ2VkIGFzIHdlIHJlYWQgaXQk">file_changed_as_we_read_it$</value>
</property>
<property>
<name>one-file-system</name>
<value>yes</value>
</property>
<property>
<name>check-device</name>
<value>no</value>
</property>
</backup-program>
</dle>
ENDDLE
To restore, position tape at start of file and run:
dd if=<tape> bs=32k
skip=1 | /bin/gzip -dc | /usr/local/libexec/amanda/application/amgtar
restore [./file-to-restore]+
1+0 records in
1+0 records out
32768 bytes (33 kB, 32 KiB) copied, 0.0179666 s, 1.8 MB/s
If I hunt down a succesfully retried dle and dump its header, there will
not be any crc reports in it.
Conclusions/clues:
1. it only blows up on a big, but randomly selected level 0
that may be from any of the 5 machines being backed up.
2. its nearly always because of a CRC error in the holding disk.
3. the holding disk has been swapped out twice now. Original was spinning
rust, 2 replacements are SSD's and are much faster.
4. amstatus always fails, IMO the failure that starts all this.
5. the error messages are not the least illuminating IMNSHO. the dle
for /GenesAmandelper/config-bak is 40Gigs but its the last 60 copys of
the configs that made these backup, and a copy of amanda's own database
for the last 60 backups, and the last 60 reports generated by amanda's
activities.
Obviously I need help. Many thanks to those who try.
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
- Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>
