Bad news. I tried to replicate the test results on my own. I downloaded and installed the first kernel marked as good: # uname -a Linux backup-01 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux I performed the tiobench and simultaneity the backup. At first it looked like the is no corruption as nothing worrying was reported in the dmesg. Unfortunately the OS behavior indicates that the there is at least memory corruption:
root@backup-01:~# dpkg-reconfigure --frontend noninteractive tzdata Can't load '/usr/lib/perl/5.18/auto/IO/IO.so' for module IO: /usr/lib/perl/5.18/auto/IO/IO.so: failed to map segment from shared object: Cannot allocate memory at /usr/share/perl/5.18/XSLoader.pm line 68. at /usr/lib/perl/5.18/IO.pm line 11. Compilation failed in require at /usr/lib/perl/5.18/IO/Handle.pm line 9. BEGIN failed--compilation aborted at /usr/lib/perl/5.18/IO/Handle.pm line 9. Compilation failed in require at /usr/lib/perl/5.18/IO/Seekable.pm line 9. BEGIN failed--compilation aborted at /usr/lib/perl/5.18/IO/Seekable.pm line 9. Compilation failed in require at /usr/lib/perl/5.18/IO/File.pm line 11. BEGIN failed--compilation aborted at /usr/lib/perl/5.18/IO/File.pm line 11. Compilation failed in require at /usr/share/perl/5.18/FileHandle.pm line 9. Compilation failed in require at /usr/share/perl5/Debconf/Template.pm line 8. BEGIN failed--compilation aborted at /usr/share/perl5/Debconf/Template.pm line 8. Compilation failed in require at /usr/share/perl5/Debconf/Question.pm line 8. BEGIN failed--compilation aborted at /usr/share/perl5/Debconf/Question.pm line 8. Compilation failed in require at /usr/share/perl5/Debconf/Config.pm line 7. BEGIN failed--compilation aborted at /usr/share/perl5/Debconf/Config.pm line 7. Compilation failed in require at /usr/share/perl5/Debconf/Log.pm line 10. Compilation failed in require at /usr/share/perl5/Debconf/Db.pm line 7. BEGIN failed--compilation aborted at /usr/share/perl5/Debconf/Db.pm line 7. Compilation failed in require at /usr/sbin/dpkg-reconfigure line 11. BEGIN failed--compilation aborted at /usr/sbin/dpkg-reconfigure line 11. I observed some random segfaults as well: [ 7773.234696] tiotest[15052]: segfault at 0 ip 000000000040183e sp 00007f2ed15a3f00 error 4 in tiotest[400000+8000] [ 7774.185489] apt-get[15073]: segfault at ffffffffffffffff ip 00007f2568a3bbfb sp 00007fff52a88840 error 5 in libapt-pkg.so.4.12.0[7f25689e5000+145000] After reboot it suddenly started to work: # dpkg-reconfigure --frontend noninteractive tzdata Current default time zone: 'SystemV/CST6CDT' Local time is now: Thu Apr 14 07:06:20 CDT 2016. Universal Time is now: Thu Apr 14 12:06:20 UTC 2016. That suggests that we're dealing with memory corruption bug that eventually leads to fs corruption. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Status in linux source package in Xenial: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts. On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here). As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt ffff88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
