Ping, any updates here?
Thanks.
On Mon, Feb 15, 2021 at 5:07 PM Ondrej Dubaj <odu...@redhat.com> wrote:
> Gentle ping
>
> On Mon, Jan 18, 2021 at 12:02 PM Ondrej Dubaj <odu...@redhat.com> wrote:
>
>> One of the customer faced I/O errors while archiving a huge file 11 TB and
>> observed that after Tar had hit read I/O error due to xfs filesystem, it
>> still continue writing 0's to the file using strace. However there was no
>> indication for tar that it was writing 0's when the error occurred.
>>
>> Later it was found that it is expected behavior to write 0's as the file
>> header is already written. Hence, it need to be padded with 0's.
>>
>> Using the reproducing steps provided by customer, we can see this behavior.
>>
>> Padding 0's is expected behavior however it does so silently (for Read error
>> at byte...), it should say it is Padding with zeros similar to how it
>> reports "File Shrank , padding with zeroes"
>>
>> During the reproducer steps provided by customer we see that sometimes tar
>> report "Read I/O errors" as "File shrank, padding with 0" , we see in the
>> step(2) provided.
>>
>> Reproducer available here:
>>
>> #!/bin/bash
>> # Reproducer "tardust"
>> #
>> # When "tar create" reads a file there are several shortcomings when it hits
>> read error
>> #
>> # 1) When read() returns 0 bytes due to read error, then this happens
>> # read(4,
>> "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 3584)
>> = 3584
>> # write(3,
>> "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 3584)
>> = 3584
>> # read(4, 0x563adef7b000, 3584) = -1 EIO (Input/output error)
>> # write(2, "tar: ", 5tar: ) = 5
>> # write(2, "/mntx/testfile: Read error at by"..., 70/mntx/testfile: Read
>> error at byte 260653056, while reading 3584 bytes) = 70
>> # write(2, ": Input/output error", 20: Input/output error) = 20
>> # write(3,
>> "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 3584)
>> = 3584
>> # write(3,
>> "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 3584)
>> = 3584
>> # write(3,
>> "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 3584)
>> = 3584
>> # Actual behaviour: it prints a message about "Read error", but it conceals
>> the information it will pad the output with zeros
>> # Expected behaviour: it should also print the information "padding with
>> zero"
>> # 2) There is a 2nd shortcoming about tar not differentiate between "read
>> error" and "file shrinkage"
>> # That means when it sees a short read due to read error, it does not report
>> read error.
>> # It looks like this:
>> # read(4,
>> "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 3584)
>> = 3584
>> # write(3,
>> "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 3584)
>> = 3584
>> # read(4,
>> "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 3584)
>> = 2560 <<< HERE
>> # write(2, "tar: ", 5tar: ) = 5
>> # write(2, "/mntx/testfile: File shrank by 5"..., 65/mntx/testfile: File
>> shrank by 53927936 bytes; padding with zeros) = 65
>> # write(2, "\n", 1
>> # ) = 1
>> # write(3,
>> "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 3584)
>> = 3584
>> # write(3,
>> "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 3584)
>> = 3584
>> # Summary: A read error is not reported here. At least it now says "padding
>> with zeros"
>> # Expected behaviour: it should report a read error, so the user knows what
>> it going on.
>> #
>> # 3) Side-Note:
>> # The blocking factor is applied to the output. When reading a file, all
>> reads are misaligned by 512 bytes.
>> # This is because it writes a 512 header for every archived file.
>> # That means the first read from file is 512bytes too short:
>> # Running with tar-blocking-factor=7
>> # fstat(1, {st_mode=S_IFREG|0644, st_size=17827, ...}) = 0
>> # write(1, "/mntx/testfile\n", 15/mntx/testfile
>> # ) = 15
>> # read(4,
>> "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 3072)
>> = 3072 #1st read 512bytes too short
>> # write(3, "mntx/testfile\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 3584) =
>> 3584
>> # read(4,
>> "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 3584)
>> = 3584
>> # write(3,
>> "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 3584)
>> = 3584
>> #
>> # 4) Reproducer overview:
>> # - Create a 500MB testimage, then create a testfile1 in the image
>> # - Use losetup/dmsetup with the "dust" target type
>> # - you can inject IO errors at specified block number in "dust"
>> # - You must hit a 4K boundary to see EIO, so use tar-blocking-factor=7 and
>> # - vary the bad blocknumber to find the case (1)
>> echo Step 1 Create disk image
>> dd if=/dev/zero of=/tmp/testimage bs=1M count=500 || exit
>> echo Step 2 Create XFS in image
>> mkfs.xfs /tmp/testimage || exit
>> echo Step 3 Use losetup so the file can be used a block device
>> losetup /dev/loop1 /tmp/testimage || exit
>> losetup
>> echo Step 6 Now create the testfile, this will have read error injected later
>> mkdir /mntx
>> mount /dev/loop1 /mntx || exit
>> dd if=/dev/zero of=/mntx/testfile bs=1M count=300 || exit
>> umount /mntx
>> echo Step7 Now iterating through bad blocks
>> echo As result, there are strace output file a1000 ... a1040
>> for i in `seq 1000 1 1040`
>> do
>> echo
>> echo Badblock $i
>> let ERR=i
>> let ERR1=i+1
>> let NUMSECTOR2=1024000-ERR1
>> #echo ERR1 is $ERR1
>> #echo NUMSECTOR2 is $NUMSECTOR2
>> dmsetup create tardust <<EOF
>> 0 $ERR linear /dev/loop1 0
>> $ERR 1 error
>> $ERR1 $NUMSECTOR2 linear /dev/loop1 $ERR1
>> EOF
>> #dmsetup ls
>> #dmsetup status
>> #dmsetup table
>> mount /dev/mapper/tardust /mntx || exit
>> strace tar cvbf 7 /tmp/tardust.tar /mntx/testfile >&/tmp/a$i
>> umount /mntx
>> dmsetup remove tardust
>> grep -e error -e shrank /tmp/a$i
>> done
>> echo "Done: inspect the strace output file for error behaviour (grep error ;
>> Look at last read()-call )"
>> losetup -d /dev/loop1
>>
>> =================
>>
>> Actual results:
>> - When tar hits a disk read error when reading file from disk and creating
>> an archive, then it prints "file shrank"
>> - then it writes zeros (aka padding) according to initial file size (but
>> does not print that message)
>> - This happens in most cases (due to tar-block-size / disk-block-size /
>> read-shift-by-512-bytes interaction)
>> - I provided a reproducer which shows under which circumstances it correctly
>> prints "Read error at byte…"
>>
>> Expected results:
>> - When there is a read error, THEN tar shall report a read error
>> - When there is a read error, THEN tar shall NOT report a "file shrank"
>> - In addition it SHALL print "Padding with zeros". This is missing currently.
>>
>>
>> Regards,
>>
>> Ondrej Dubaj
>>
>>
