We're seeing errors very similar to this on 2 16.04 VMs, both attached to different SANs. Sadly one is the live data and the other the backup, so...
kern.log snippet: Apr 25 14:54:52 oaebackup1 kernel: [11778105.146482] EXT4-fs (sda1): previous I/O error to superblock detected Apr 25 14:54:52 oaebackup1 kernel: [11778105.151094] sd 6:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Apr 25 14:54:52 oaebackup1 kernel: [11778105.151102] sd 6:0:0:0: [sda] tag#0 Sense Key : Data Protect [current] Apr 25 14:54:52 oaebackup1 kernel: [11778105.151115] sd 6:0:0:0: [sda] tag#0 Add. Sense: Space allocation failed write protect Apr 25 14:54:52 oaebackup1 kernel: [11778105.151120] sd 6:0:0:0: [sda] tag#0 CDB: Write(16) 8a 08 00 00 00 00 00 00 08 00 00 00 00 08 00 00 Apr 25 14:54:52 oaebackup1 kernel: [11778105.151124] blk_update_request: critical space allocation error, dev sda, sector 2048 Apr 25 14:54:52 oaebackup1 kernel: [11778105.154859] Buffer I/O error on dev sda1, logical block 0, lost sync page write Apr 25 14:54:52 oaebackup1 kernel: [11778105.158519] EXT4-fs error (device sda1): ext4_readdir:224: inode #11: block 11595: comm bash: path /mnt/oae-backups/lost+found: bad entry in directoryr0 Apr 25 14:54:52 oaebackup1 kernel: [11778105.165899] EXT4-fs (sda1): previous I/O error to superblock detected Apr 25 14:54:52 oaebackup1 kernel: [11778105.170487] sd 6:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Apr 25 14:54:52 oaebackup1 kernel: [11778105.170494] sd 6:0:0:0: [sda] tag#0 Sense Key : Data Protect [current] Apr 25 14:54:52 oaebackup1 kernel: [11778105.170503] sd 6:0:0:0: [sda] tag#0 Add. Sense: Space allocation failed write protect Apr 25 14:54:52 oaebackup1 kernel: [11778105.170511] sd 6:0:0:0: [sda] tag#0 CDB: Write(16) 8a 08 00 00 00 00 00 00 08 00 00 00 00 08 00 00 Apr 25 14:54:52 oaebackup1 kernel: [11778105.170515] blk_update_request: critical space allocation error, dev sda, sector 2048 Apr 25 14:54:52 oaebackup1 kernel: [11778105.174225] Buffer I/O error on dev sda1, logical block 0, lost sync page write Apr 25 14:55:26 oaebackup1 kernel: [11778139.432404] EXT4-fs error: 851 callbacks suppressed Apr 25 14:55:26 oaebackup1 kernel: [11778139.432430] EXT4-fs error (device sda1): ext4_lookup:1583: inode #77009420: comm ls: deleted inode referenced: 77531011 Apr 25 14:55:26 oaebackup1 kernel: [11778139.436785] EXT4-fs: 851 callbacks suppressed Apr 25 14:55:26 oaebackup1 kernel: [11778139.436789] EXT4-fs (sda1): previous I/O error to superblock detected Apr 25 14:55:26 oaebackup1 kernel: [11778139.442236] scsi_io_completion: 851 callbacks suppressed Apr 25 14:55:26 oaebackup1 kernel: [11778139.442248] sd 6:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Apr 25 14:55:26 oaebackup1 kernel: [11778139.442255] sd 6:0:0:0: [sda] tag#0 Sense Key : Data Protect [current] Apr 25 14:55:26 oaebackup1 kernel: [11778139.442260] sd 6:0:0:0: [sda] tag#0 Add. Sense: Space allocation failed write protect Apr 25 14:55:26 oaebackup1 kernel: [11778139.442266] sd 6:0:0:0: [sda] tag#0 CDB: Write(16) 8a 08 00 00 00 00 00 00 08 00 00 00 00 08 00 00 Apr 25 14:55:26 oaebackup1 kernel: [11778139.442268] blk_update_request: 851 callbacks suppressed Apr 25 14:55:26 oaebackup1 kernel: [11778139.442271] blk_update_request: critical space allocation error, dev sda, sector 2048 Apr 25 14:55:26 oaebackup1 kernel: [11778139.446366] buffer_io_error: 851 callbacks suppressed Apr 25 14:55:26 oaebackup1 kernel: [11778139.446370] Buffer I/O error on dev sda1, logical block 0, lost sync page write Apr 25 14:55:26 oaebackup1 kernel: [11778139.450566] EXT4-fs error (device sda1): ext4_lookup:1583: inode #77009420: comm ls: deleted inode referenced: 77531009 Apr 25 14:55:26 oaebackup1 kernel: [11778139.454853] EXT4-fs (sda1): previous I/O error to superblock detected Apr 25 14:55:26 oaebackup1 kernel: [11778139.460198] sd 6:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Apr 25 14:55:26 oaebackup1 kernel: [11778139.460206] sd 6:0:0:0: [sda] tag#0 Sense Key : Data Protect [current] Apr 25 14:55:26 oaebackup1 kernel: [11778139.460210] sd 6:0:0:0: [sda] tag#0 Add. Sense: Space allocation failed write protect Apr 25 14:55:26 oaebackup1 kernel: [11778139.460215] sd 6:0:0:0: [sda] tag#0 CDB: Write(16) 8a 08 00 00 00 00 00 00 08 00 00 00 00 08 00 00 Apr 25 14:55:26 oaebackup1 kernel: [11778139.460219] blk_update_request: critical space allocation error, dev sda, sector 2048 Apr 25 14:55:26 oaebackup1 kernel: [11778139.464438] Buffer I/O error on dev sda1, logical block 0, lost sync page write Apr 25 14:55:26 oaebackup1 kernel: [11778139.468425] EXT4-fs error (device sda1): ext4_lookup:1583: inode #77009420: comm ls: deleted inode referenced: 77531010 Apr 25 14:55:26 oaebackup1 kernel: [11778139.472349] EXT4-fs (sda1): previous I/O error to superblock detected Apr 25 14:55:26 oaebackup1 kernel: [11778139.476925] sd 6:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Kernel: # uname -a Linux oaedata1 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1418339 Title: kernel set the wrong max_sectors_kb for 4K disks(sd.c) Status in linux package in Ubuntu: Fix Released Bug description: [Impact] Problem Description =========================== We get following error messages on the console and IO fails. [ 63.322503] sd 0:2:2:0: [sdc] [ 63.322603] Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE [ 63.322661] sd 0:2:2:0: [sdc] [ 63.322697] Sense Key : Illegal Request [current] [ 63.322757] sd 0:2:2:0: [sdc] [ 63.322792] Add. Sense: Invalid field in cdb [ 63.322839] sd 0:2:2:0: [sdc] CDB: [ 63.322874] Write(10): 2a 00 03 02 17 a0 00 00 80 00 [ 63.323038] blk_update_request: critical target error, dev sdc, sector 403750144 [ 63.323117] sd 0:2:2:0: [sdc] [ 63.323152] Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE [ 63.323210] sd 0:2:2:0: [sdc] [ 63.323245] Sense Key : Illegal Request [current] [ 63.323303] sd 0:2:2:0: [sdc] [ 63.323338] Add. Sense: Invalid field in cdb [ 63.323385] sd 0:2:2:0: [sdc] CDB: [ 63.323420] Write(10): 2a 00 03 02 17 20 00 00 80 00 [ 63.323580] blk_update_request: critical target error, dev sdc, sector 403749120 [ 138.410612] EXT4-fs warning (device dm-2): ext4_end_bio:317: I/O error -121 writing to inode 265776 (offset 0 size 393216 starting block 33920) [ 138.410727] Buffer I/O error on device dm-2, logical block 33920 [ 138.410785] Buffer I/O error on device dm-2, logical block 33921 [ 138.410842] Buffer I/O error on device dm-2, logical block 33922 [ 138.410900] Buffer I/O error on device dm-2, logical block 33923 [ 138.410957] Buffer I/O error on device dm-2, logical block 33924 ---uname output--- Ubuntu 15.04 Machine Type = P8(Alpine) [Test Case] Steps to Reproduce ===================== read/write to a 4K array. [Fix] commit 3a9794d32984b67a6d8992226918618f0e51e5d5 upstream To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1418339/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
