Package: linux-image-4.15.0-3-amd64
Version: 4.15.17-1
It looks like on a larger ext3 file system when writing new files and some
conditions are met, the ext4 subsystem thinks the fs is corrupted.
I haven't encountered any problems with smaller fs's yet ( / is usually
less than 20 GB on my machines and no problems on such partitions yet).
I don't have similarly large "real" ext4 file systems, only this ext3, so I
couldn't test it if it is really ext3 only.
The raw SMART values of my raid1 array's disks are OK, I also did a few
passes with "echo check > /sys/block/md1/md/sync_action" just to be sure
none of the disks corrupted my data silently (each pass took around 4
hours).
I'll leave some sample error messages at the end of this report, as they
are rather long and repeating. Basically they say:
EXT4-fs error (device md1): ext4_validate_block_bitmap:401: comm <some
program's name>: bg <smaller number>: block <larger number>: invalid block
bitmap
On 4.15.0-3, extracting some rar files where the extracted data should be
below 1 GB, the free space (hundreds of Gbytes) disappeared quickly, and
dmesg showed the above error message repeating, so I had to kill the rar
process. The same happens if I scp files to it; basically anything that
results in writes can trigger it. If I umount and run e2fsck, it only finds
that the free space count is wrong and fixes it. If the system is running
on 4.15.0-3, the error is repeatable after reboots, poweroffs.
With the previous 4.15.0-2 (4.15.11-1) kernel, the repaired file system can
be used without errors, so I'm using that for the time being.
The fs in question was created as ext3 some years ago; on boot the kernels
print this which is expected and normal:
EXT4-fs (md1): mounting ext3 file system using the ext4 subsystem
EXT4-fs (md1): mounted filesystem with ordered data mode. Opts: acl
Information from /proc/mdstat :
md1 : active raid1 sda3[0] sdb3[1]
1926249424 blocks super 1.2 [2/2] [UU]
df -T:
Filesystem Type 1K-blocks Used Available Use% Mounted on
/dev/md1 ext3 1910942232 1521788612 389153620 80% /storage
I think the error is in fs/ext4/balloc.c , but I can be mistaken - the
error message comes from there. This file received a patch recently:
ext4-add-validity-checks-for-bitmap-block-numbers.patch , but skimming over
it I don't see what it could have changed to cause the errors - although
I'm no kernel developer.
About the filesystem from the head of dumpe2fs (notice, the inode size is
128 instead of the more common 256, and sparse_super was used; back then I
liked to use these options for various reasons):
dumpe2fs 1.44.1 (24-Mar-2018)
Filesystem volume name: HPg8storage
Last mounted on: /storage
Filesystem UUID: 0275a7f2-53ea-48fb-8c32-a3a9b0c25406
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index
filetype sparse_super large_file
Filesystem flags: signed_directory_hash
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 120397824
Block count: 481562356
Reserved block count: 0
Free blocks: 97288373
Free inodes: 116239195
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 909
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 256
RAID stride: 1
RAID stripe width: 1
Filesystem created: Fri Feb 13 19:39:29 2015
Last mount time: Wed May 2 21:05:27 2018
Last write time: Wed May 2 21:07:37 2018
Mount count: 1
Maximum mount count: -1
Last checked: Wed May 2 20:46:29 2018
Check interval: 15552000 (6 months)
Next check after: Mon Oct 29 19:46:29 2018
Lifetime writes: 1867 GB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 128
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: 7f682190-7662-445e-9e5d-ff0fc9ab3104
Journal backup: inode blocks
Journal features: journal_incompat_revoke
Journal size: 128M
Journal length: 32768
Journal sequence: 0x000211a9
Journal start: 0
A sample of the error messages from kernel.log:
May 1 12:45:57 hpg8 kernel: [15952.265682] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5289: block 173315497: invalid
block bitmap
May 1 12:45:57 hpg8 kernel: [15952.330241] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5290: block 173348266: invalid
block bitmap
May 1 12:45:57 hpg8 kernel: [15952.363247] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5291: block 173381035: invalid
block bitmap
May 1 12:45:57 hpg8 kernel: [15952.388234] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5292: block 173413804: invalid
block bitmap
May 1 12:45:57 hpg8 kernel: [15952.427087] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5293: block 173446573: invalid
block bitmap
May 1 12:45:57 hpg8 kernel: [15952.456162] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5294: block 173479342: invalid
block bitmap
May 1 12:45:57 hpg8 kernel: [15952.494708] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5295: block 173512111: invalid
block bitmap
May 1 12:45:57 hpg8 kernel: [15952.525483] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5296: block 173544880: invalid
block bitmap
May 1 12:45:57 hpg8 kernel: [15952.558537] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5297: block 173577649: invalid
block bitmap
May 1 12:45:57 hpg8 kernel: [15952.594656] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5298: block 173610418: invalid
block bitmap
May 1 12:46:02 hpg8 kernel: [15957.283883] EXT4-fs error: 137 callbacks
suppressed
May 1 12:46:02 hpg8 kernel: [15957.283886] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5436: block 178132540: invalid
block bitmap
May 1 12:46:02 hpg8 kernel: [15957.313168] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5437: block 178165309: invalid
block bitmap
May 1 12:46:02 hpg8 kernel: [15957.363793] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5438: block 178198078: invalid
block bitmap
May 1 12:46:02 hpg8 kernel: [15957.413221] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5439: block 178230847: invalid
block bitmap
May 1 12:46:02 hpg8 kernel: [15957.443555] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5440: block 178263616: invalid
block bitmap
May 1 12:46:02 hpg8 kernel: [15957.477881] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5441: block 178296385: invalid
block bitmap
May 1 12:46:02 hpg8 kernel: [15957.512082] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5442: block 178329154: invalid
block bitmap
May 1 12:46:02 hpg8 kernel: [15957.552511] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5443: block 178361923: invalid
block bitmap
May 1 12:46:02 hpg8 kernel: [15957.581757] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5444: block 178394692: invalid
block bitmap
May 1 12:46:02 hpg8 kernel: [15957.619541] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5445: block 178427461: invalid
block bitmap
May 1 12:46:07 hpg8 kernel: [15962.304781] EXT4-fs error: 136 callbacks
suppressed
May 1 12:46:07 hpg8 kernel: [15962.304783] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5582: block 182916814: invalid
block bitmap
May 1 12:46:07 hpg8 kernel: [15962.342431] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5583: block 182949583: invalid
block bitmap
May 1 12:46:07 hpg8 kernel: [15962.383918] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5584: block 182982352: invalid
block bitmap
May 1 12:46:07 hpg8 kernel: [15962.441696] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5585: block 183015121: invalid
block bitmap
May 1 12:46:07 hpg8 kernel: [15962.476039] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5586: block 183047890: invalid
block bitmap
May 1 12:46:07 hpg8 kernel: [15962.516458] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5587: block 183080659: invalid
block bitmap
May 1 12:46:07 hpg8 kernel: [15962.544928] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5588: block 183113428: invalid
block bitmap
May 1 12:46:07 hpg8 kernel: [15962.582115] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5589: block 183146197: invalid
block bitmap
May 1 12:46:07 hpg8 kernel: [15962.612453] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5590: block 183178966: invalid
block bitmap
May 1 12:46:07 hpg8 kernel: [15962.648108] EXT4-fs error (device md1):
ext4_validate_block_bitmap:401: comm rar: bg 5591: block 183211735: invalid
block bitmap
...
