Hi again.
So, as expected, the warning occurred again when job 94511
was copied (but also two new warnings for this specific job):
Warning: dird/catreq.cc:608 MD5 digest not same File=3 as
attributes=2
Warning: dird/catreq.cc:608 MD5 digest not same File=18 as
attributes=17
Warning: dird/catreq.cc:608 MD5 digest not same File=23 as
attributes=22
The restore of the copy also errored in the same manner as
before:
2024-02-26 10:43:50 bareos-fd JobId 95784: Error:
findlib/attribs.cc:381 File size of restored file
/tmp/bareos-restores/var/backup/old/2024-02-07_23:30/backup_www_1100CC.tar.gz
not correct. Original 738212473, restored 442105856.
2024-02-26 10:44:10 bareos-fd JobId 95784: Error:
findlib/attribs.cc:381 File size of restored file
/tmp/bareos-restores/var/backup/old/2024-02-12_23:30/backup_www_1100CC.tar.gz
not correct. Original 856739265, restored 783548416.
2024-02-26 10:44:17 bareos-fd JobId 95784: Error:
findlib/attribs.cc:381 File size of restored file
/tmp/bareos-restores/var/backup/old/2024-02-14_23:30/backup_www_1100CC.tar.gz
not correct. Original 856739984, restored 31064064.
For now I'll try disabling autoxflate and see if that solves
the issue.
Best regards,
Samuel
On Friday, February 23, 2024 at 2:02:28 PM UTC+1 Samuel wrote:
Thanks for looking into it!
Hm, there seems to be no file with fileindex=3 in the
database for the copy:
Enter SQL query: select * from file where jobid=94146 and
fileindex=3;
No results to list.
All files that are being backed up are just regular
files, nothing special.
The copy runs locally on the director's host from its SD
to the same SD.
Here's file 2 and 3 of today's consolidated incremental
backup which will be copied tomorrow and most likely lead
to the same warning again as it has for the last few days
(for this specific fileset it always warns about `File=3
attributes=2`):
Enter SQL query: select *, decode_lstat(lstat) from file
where jobid=94511 and fileindex=2;
+-------------+-----------+-------+---------+----------+--------+--------+--------+----------------------------------------------------------------+------------------------+--------------------------+----------------------------------------------------------------------------------------------+
| fileid | fileindex | jobid | pathid | deltaseq |
markid | fhinfo | fhnode | lstat | md5
| name | decode_lstat
|
+-------------+-----------+-------+---------+----------+--------+--------+--------+----------------------------------------------------------------+------------------------+--------------------------+----------------------------------------------------------------------------------------------+
| 484,575,603 | 2 | 94511 | 910,901 | 0 |
0 | 0 | 0 | P0D CgA4 IGk B A i A sADp5 BAA
FgAo BlxAt1 BlxASF BlxVYH A A d | D/VouduY5TF4KFacjRE7Hw
| backup_www_1100CC.tar.gz |
(64771,655416,33188,1,0,34,0,738212473,4096,1441832,1707346805,1707345029,1707431431,0,0,29)
|
+-------------+-----------+-------+---------+----------+--------+--------+--------+----------------------------------------------------------------+------------------------+--------------------------+----------------------------------------------------------------------------------------------+
Enter SQL query: select *, decode_lstat(lstat) from file
where jobid=94511 and fileindex=3;
+-------------+-----------+-------+---------+----------+--------+--------+--------+---------------------------------------------------------------+------------------------+-----------------------------+-------------------------------------------------------------------------------------------+
| fileid | fileindex | jobid | pathid | deltaseq |
markid | fhinfo | fhnode | lstat | md5
| name |
decode_lstat |
+-------------+-----------+-------+---------+----------+--------+--------+--------+---------------------------------------------------------------+------------------------+-----------------------------+-------------------------------------------------------------------------------------------+
| 484,575,604 | 3 | 94511 | 910,901 | 0 |
0 | 0 | 0 | P0D CgA5 IGk B A i A BA9gA BAA
IHw BlxAt8 BlxASH BlxVYH A A d | J9h+OdR/6XCqhqpJdJvYtw |
backup_databases_1100CC.tar |
(64771,655417,33188,1,0,34,0,17029120,4096,33264,1707346812,1707345031,1707431431,0,0,29)
|
+-------------+-----------+-------+---------+----------+--------+--------+--------+---------------------------------------------------------------+------------------------+-----------------------------+-------------------------------------------------------------------------------------------+
Restore of this job (jobid=94511) works just fine:
23-Feb 13:44 bareos-sd JobId 94559: Releasing device
"FileDevice-ReadOnly-0005" (/backup_1/bareos).
23-Feb 13:44 bareos-dir JobId 94559: Max configured use
duration=82,800 sec. exceeded. Marking Volume
"AI-Consolidated-1771" as Used.
23-Feb 13:44 bareos-dir JobId 94559: Bareos bareos-dir
23.0.2~pre32.0a0e55739 (31Jan24):
Build OS: Ubuntu 20.04.5 LTS
JobId: 94559
Job: Restore-Files.2024-02-23_13.43.47_38
Restore Client: "bareos-fd"
23.0.2~pre32.0a0e55739 (31Jan24) Ubuntu 20.04.5 LTS,ubuntu
Start time: 23-Feb-2024 13:43:49
End time: 23-Feb-2024 13:44:20
Elapsed time: 31 secs
Files Expected: 33
Files Restored: 33
Bytes Restored: 9,143,514,060
Rate: 294952.1 KB/s
FD Errors: 0
FD termination status: OK
SD termination status: OK
Bareos binary info: Bareos community build
(UNSUPPORTED): Get professional support from
https://www.bareos.com
Job triggered by: User
Termination: Restore OK
I'll also try restoring its copy from tape once it has
been copied.
Best regards,
Samuel
On Friday, February 23, 2024 at 1:16:32 PM UTC+1
Sebastian Sura wrote:
I meant to say File 3 has no stream 1, sorry for the
confusion!
Am 23.02.24 um 13:15 schrieb Sebastian Sura:
Hi Samuel
thanks for the gathering this info. Ill look into
the bscan issue as well though i think this might be
a known issue.
Regardless, regarding your actual issue: The bscan
output shows that File 3 is missing its attributes
(it has no stream 3).
This confuses the director as it never got told that
file 3 now started getting backed up and this is why
you get the warning
message from the director.
The same is happening during the restore: the
filedaemon never got told that now a new file was
started (because it never got the attribute stream)
so it basically merged both File 2 and File 3 into
one file. Even with this in mind, i think the
restore should have caught that and issues a warning.
Ill look into why this did not happen.
If you add up all the data records (stream=2) with
fileid=2 and fileid=3, you will get 738212473 bytes,
which is exactly what the filedaemon reported
as size for file 2. Since the log contains the size
it expected i imagine that you could manually
restore file 3 by splitting file 2 into two.
Can you check which file fileid 3 corresponds to ?
Is it a special kind of file or just a normal one ?
Ill try to see if i can reproduce your issue in the
copy system test. Do you do a local copy (so copy
to the same sd) or a remote one?
Kind Regards
Sebastian Sura
Am 23.02.24 um 12:56 schrieb 'Samuel' via bareos-users:
Hi Sebastian,
After looking at the logs some more I noticed that
all jobs with this warning have in common that
they're using compression (LZ4) and autoxflate on
replication.
Perhaps this combination is still not entirely
fixed yet. I'll try disabling autoxflate in the
coming days.
The entire result of bscan is in the attachment.
However here's the last part of it where bscan
seems to abort (?):
$ sudo -u bareos bscan -b
md5_digest_error_copy_only.bsr --list-records
TapeDevice2 2>&1 | tee records.txt
...
bscan: stored/bscan.cc:494-0 Record: SessId=3480
SessTim=1707222936 FileIndex=23 Stream=2 len=65536
bscan: stored/bscan.cc:494-0 Record: SessId=3480
SessTim=1707222936 FileIndex=23 Stream=2 len=65536
bscan: stored/bscan.cc:494-0 Record: SessId=3480
SessTim=1707222936 FileIndex=23 Stream=2 len=65536
bscan: stored/bscan.cc:494-0 Record: SessId=3480
SessTim=1707222936 FileIndex=23 Stream=2 len=22528
bscan: stored/bscan.cc:494-0 Record: SessId=3480
SessTim=1707222936 FileIndex=23 Stream=3 len=16
bscan: stored/bscan.cc:494-0 Record: SessId=3480
SessTim=1707222936 FileIndex=24 Stream=1 len=100
bscan: stored/bscan.cc:494-0 Record: SessId=3480
SessTim=1707222936 FileIndex=25 Stream=1 len=83
bscan: stored/bscan.cc:494-0 Record: SessId=3480
SessTim=1707222936 FileIndex=-5 Stream=94145 len=193
bscan: stored/bscan.cc:681-0 Could not find
SessId=3480 SessTime=1707222936 for EOS record.
Records would have been added or updated in the
catalog:
0 Media
1 Pool
0 Job
0 File
0 RestoreObject
23-Feb 12:05 bscan JobId 0: Releasing device
"TapeDevice2"
(/dev/tape/by-id/scsi-35000e111c71ac0bf-nst).
I also tried restoring the copy which ends in error.
Most files are restored successfully except for one
file which happens to have FileIndex=2
(corresponding to the `attributes=2` in the warning?):
Enter SQL query: select path,name from file,path
where jobid=94146 and fileindex=2 and
file.pathid=path.pathid;
+-----------------------------------+--------------------------+
| path | name
|
+-----------------------------------+--------------------------+
| /var/backup/old/2024-02-07_23:30/ |
backup_www_1100CC.tar.gz |
+-----------------------------------+--------------------------+
A full restore of the original non-copy backup jobs
works fine, as do restores of other copy jobs on
tape of filesets that don't use compression.
The original job that was copied (93682) doesn't
exist anymore; I think it was an always-incremental
consolidated incremental backup which got
consolidated again the following day.
Best regards,
Samuel
--
You received this message because you are
subscribed to the Google Groups "bareos-users" group.
To unsubscribe from this group and stop receiving
emails from it, send an email to
bareos-users...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/bareos-users/e766e8c6-5b58-424f-816f-f7c5d7252180n%40googlegroups.com
<https://groups.google.com/d/msgid/bareos-users/e766e8c6-5b58-424f-816f-f7c5d7252180n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
Sebastian surasebasti...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221 630693-0
https://www.bareos.com
Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: Stephan Dühr, Jörg Steffens, Philipp Storz
--
You received this message because you are subscribed
to the Google Groups "bareos-users" group.
To unsubscribe from this group and stop receiving
emails from it, send an email to
bareos-users...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/bareos-users/111cecfe-d3e1-4eb9-8a0e-f053a2777d0f%40bareos.com
<https://groups.google.com/d/msgid/bareos-users/111cecfe-d3e1-4eb9-8a0e-f053a2777d0f%40bareos.com?utm_medium=email&utm_source=footer>.
--
Sebastian surasebasti...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221 630693-0
https://www.bareos.com
Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: Stephan Dühr, Jörg Steffens, Philipp Storz
--
You received this message because you are subscribed to the
Google Groups "bareos-users" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to bareos-users...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/bareos-users/3823a22d-b49e-4892-8b45-6c932ce7e699n%40googlegroups.com
<https://groups.google.com/d/msgid/bareos-users/3823a22d-b49e-4892-8b45-6c932ce7e699n%40googlegroups.com?utm_medium=email&utm_source=footer>.
