[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
How about rusty? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: In Progress Status in linux source package in Xenial: Fix Committed Status in linux source package in Yakkety: Fix Committed Status in linux source package in Zesty: Fix Released Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
3.13.0-103-generic #150~lp1470250 SMP looks promising. Several days without crash. I encountered some IO errors but the filesystem remains rw. [Wed Jan 11 13:47:35 2017] hv_storvsc vmbus_0_1: cmd 0x35 scsi status 0x2 srb status 0x82 [Wed Jan 11 13:47:35 2017] hv_storvsc vmbus_0_1: stor pkt 8800f0b0f728 autosense data valid - len 18 [Wed Jan 11 13:47:35 2017] storvsc: Sense Key : Unit Attention [current] [Wed Jan 11 13:47:35 2017] storvsc: Add. Sense: Changed operating definition [Wed Jan 11 13:47:35 2017] sd 2:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. [Wed Jan 11 13:47:35 2017] end_request: I/O error, dev sda, sector 0 [Wed Jan 11 13:48:56 2017] hv_storvsc vmbus_0_1: cmd 0x28 scsi status 0x2 srb status 0x82 [Wed Jan 11 13:48:56 2017] hv_storvsc vmbus_0_1: stor pkt 8800f0fa0668 autosense data valid - len 18 [Wed Jan 11 13:48:56 2017] storvsc: Sense Key : Unit Attention [current] [Wed Jan 11 13:48:56 2017] storvsc: Add. Sense: Changed operating definition [Wed Jan 11 13:48:56 2017] sd 2:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. [Wed Jan 11 14:07:56 2017] hv_storvsc vmbus_0_1: cmd 0x2a scsi status 0x2 srb status 0x82 [Wed Jan 11 14:07:56 2017] hv_storvsc vmbus_0_1: stor pkt 8800f0fa50a8 autosense data valid - len 18 [Wed Jan 11 14:07:56 2017] storvsc: Sense Key : Unit Attention [current] [Wed Jan 11 14:07:56 2017] storvsc: Add. Sense: Changed operating definition [Wed Jan 11 14:07:56 2017] sd 2:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Yakkety: In Progress Status in linux source package in Zesty: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~ker
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
I can confirm that the trusty patches are preventing the filesystem failure. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Yakkety: In Progress Status in linux source package in Zesty: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
I also found someone reporting the issue on RedHat: https://social.technet.microsoft.com/Forums/office/en-US/cfe15e32-bfbc-47e0-8d2b-382a1293b9aa/vss-issues-with-centos-66-x64?forum=linuxintegrationservices >From my tests it looked like Centos kernels are not affected but above report casts some shadow on that assumption. It might be that the buggy code was integrated into RH kernels as well. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Lo luck here :( Even though I have the hv_vss_daemon that logs FREEZE/THAW I experienced crashes: Mar 15 10:36:38 emsi-02 Hyper-V VSS: VSS: op=FREEZE: succeeded Mar 15 10:36:38 emsi-02 kernel: [320343.560935] sd 2:0:0:0: [storvsc] Sense Key : Unit Attention [current] Mar 15 10:36:38 emsi-02 kernel: [320343.560949] sd 2:0:0:0: [storvsc] Add. Sense: Changed operating definition Mar 15 10:36:38 emsi-02 kernel: [320343.560958] sd 2:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Mar 15 10:36:38 emsi-02 Hyper-V VSS: VSS: op=THAW: succeeded Mar 15 10:37:08 emsi-02 kernel: [320373.536208] sd 2:0:0:0: [storvsc] Sense Key : Unit Attention [current] Mar 15 10:37:08 emsi-02 kernel: [320373.536222] sd 2:0:0:0: [storvsc] Add. Sense: Changed operating definition Mar 15 10:37:08 emsi-02 kernel: [320373.536287] sd 2:0:0:0: Warning! Received an indication that the operating parameters on thi and then later on: [Wed Mar 16 10:06:45 2016] sd 2:0:0:0: [storvsc] Sense Key : Unit Attention [current] [Wed Mar 16 10:06:45 2016] sd 2:0:0:0: [storvsc] Add. Sense: Changed operating definition [Wed Mar 16 10:06:45 2016] sd 2:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. [Wed Mar 16 10:08:12 2016] sd 2:0:0:0: [storvsc] Sense Key : Unit Attention [current] [Wed Mar 16 10:08:12 2016] sd 2:0:0:0: [storvsc] Add. Sense: Changed operating definition [Wed Mar 16 10:08:12 2016] sd 2:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. [Wed Mar 16 10:08:12 2016] blk_update_request: I/O error, dev sda, sector 1061736 [Wed Mar 16 10:08:12 2016] Aborting journal on device sda1-8. [Wed Mar 16 10:08:12 2016] EXT4-fs error (device sda1): ext4_journal_check_start:56: [Wed Mar 16 10:08:12 2016] EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal [Wed Mar 16 10:08:12 2016] EXT4-fs (sda1): Remounting filesystem read-only [Wed Mar 16 10:08:12 2016] EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal [Wed Mar 16 10:08:12 2016] EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal [Wed Mar 16 10:08:12 2016] Detected aborted journal [Wed Mar 16 10:08:12 2016] EXT4-fs error (device sda1) in ext4_writepages:2421: IO failure -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
It looks like there is a race condition and the corruption occurs when the FREEZE happens to late. No wonder as it's happening from userland. Mar 16 15:06:49 emsi-02 kernel: [422953.038989] sd 2:0:0:0: [storvsc] Sense Key : Unit Attention [current] Mar 16 15:06:49 emsi-02 kernel: [422953.039003] sd 2:0:0:0: [storvsc] Add. Sense: Changed operating definition Mar 16 15:06:49 emsi-02 kernel: [422953.039058] sd 2:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Mar 16 15:06:49 emsi-02 kernel: [422953.039063] blk_update_request: I/O error, dev sda, sector 0 Mar 16 15:06:49 emsi-02 Hyper-V VSS: VSS: op=FREEZE: succeeded Mar 16 15:06:49 emsi-02 kernel: [422953.072223] sd 2:0:0:0: [storvsc] Sense Key : Unit Attention [current] Mar 16 15:06:49 emsi-02 kernel: [422953.072235] sd 2:0:0:0: [storvsc] Add. Sense: Changed operating definition Mar 16 15:06:49 emsi-02 kernel: [422953.072242] sd 2:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Mar 16 15:06:49 emsi-02 Hyper-V VSS: VSS: op=THAW: succeeded -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
It's a regression. Anyone knows the last working kernel version? It might be easier to find the breaking change. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Now I got much more verbose error: [Wed Mar 30 11:38:59 2016] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 25s! [init:1321] [Wed Mar 30 11:38:59 2016] Modules linked in: btrfs(E) xor(E) raid6_pq(E) ufs(E) qnx4(E) hfsplus(E) hfs(E) minix(E) ntfs(E) msdos(E) jfs(E) xfs(E) libcrc32c(E) xt_nat xt_tcpudp veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables x_tables nf_nat nf_conntrack br_netfilter bridge stp llc aufs sit tunnel4 ip_tunnel i2c_piix4 input_leds serio_raw hyperv_fb hyperv_keyboard 8250_fintek joydev mac_hid autofs4 hid_generic hv_netvsc hid_hyperv hv_utils hid hv_storvsc crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse pata_acpi hv_vmbus floppy [Wed Mar 30 11:38:59 2016] CPU: 1 PID: 1321 Comm: init Tainted: GE 4.2.0-30-generic #35-Ubuntu [Wed Mar 30 11:38:59 2016] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090006 05/23/2012 [Wed Mar 30 11:38:59 2016] task: 88003585b700 ti: 8800f3bb task.ti: 8800f3bb [Wed Mar 30 11:38:59 2016] RIP: 0010:[] [] __build_skb+0xbe/0x100 [Wed Mar 30 11:38:59 2016] RSP: 0018:880102643c08 EFLAGS: 00010286 [Wed Mar 30 11:38:59 2016] RAX: RBX: RCX: [Wed Mar 30 11:38:59 2016] RDX: 8800ee0ea400 RSI: 0020 RDI: 8800f08221c0 [Wed Mar 30 11:38:59 2016] RBP: 880102643c18 R08: 03c0 R09: 8800ee0ea400 [Wed Mar 30 11:38:59 2016] R10: 0003 R11: 0004 R12: 880102643b78 [Wed Mar 30 11:38:59 2016] R13: 817f4a9b R14: 880102643c18 R15: 0600 [Wed Mar 30 11:38:59 2016] FS: 7ffa15ff9800() GS:88010264() knlGS: [Wed Mar 30 11:38:59 2016] CS: 0010 DS: ES: CR0: 80050033 [Wed Mar 30 11:38:59 2016] CR2: 7f1407ee4000 CR3: 35929000 CR4: 001406e0 [Wed Mar 30 11:38:59 2016] Stack: [Wed Mar 30 11:38:59 2016] 0500 88010115c000 880102643c58 816d7615 [Wed Mar 30 11:38:59 2016] 03ff 88010115c000 880102643d38 c90001b70c78 [Wed Mar 30 11:38:59 2016] 880035158400 880102660a48 880102643ca8 c012fbae [Wed Mar 30 11:38:59 2016] Call Trace: [Wed Mar 30 11:38:59 2016] [Wed Mar 30 11:38:59 2016] [] __netdev_alloc_skb+0xc5/0x110 [Wed Mar 30 11:38:59 2016] [] netvsc_recv_callback+0x6e/0x150 [hv_netvsc] [Wed Mar 30 11:38:59 2016] [] rndis_filter_receive+0x131/0x5b0 [hv_netvsc] [Wed Mar 30 11:38:59 2016] [] ? hv_copyfrom_ringbuffer+0x46/0x90 [hv_vmbus] [Wed Mar 30 11:38:59 2016] [] netvsc_receive+0x114/0x1f0 [hv_netvsc] [Wed Mar 30 11:38:59 2016] [] ? vmbus_recvpacket_raw+0x9b/0xe0 [hv_vmbus] [Wed Mar 30 11:38:59 2016] [] netvsc_channel_cb+0x190/0x3f0 [hv_netvsc] [Wed Mar 30 11:38:59 2016] [] ? sched_clock_local+0x1b/0x90 [Wed Mar 30 11:38:59 2016] [] vmbus_on_event+0x154/0x190 [hv_vmbus] [Wed Mar 30 11:38:59 2016] [] tasklet_action+0xdf/0x100 [Wed Mar 30 11:38:59 2016] [] __do_softirq+0xf6/0x250 [Wed Mar 30 11:38:59 2016] [] irq_exit+0xa3/0xb0 [Wed Mar 30 11:38:59 2016] [] hyperv_vector_handler+0x42/0x60 [Wed Mar 30 11:38:59 2016] [] hyperv_callback_vector+0x6b/0x70 [Wed Mar 30 11:38:59 2016] [Wed Mar 30 11:38:59 2016] [] ? __aa_path_perm+0xf4/0x110 [Wed Mar 30 11:38:59 2016] [] ? copy_user_enhanced_fast_string+0x5/0x10 [Wed Mar 30 11:38:59 2016] [] ? set_fd_set+0x21/0x30 [Wed Mar 30 11:38:59 2016] [] core_sys_select+0x202/0x2f0 [Wed Mar 30 11:38:59 2016] [] ? cp_new_stat+0x156/0x190 [Wed Mar 30 11:38:59 2016] [] ? seccomp_phase1+0x98/0x290 [Wed Mar 30 11:38:59 2016] [] ? read_hv_clock_tsc+0x3c/0x90 [hv_vmbus] [Wed Mar 30 11:38:59 2016] [] ? ktime_get_ts64+0x45/0xf0 [Wed Mar 30 11:38:59 2016] [] SyS_select+0xba/0x110 [Wed Mar 30 11:38:59 2016] [] entry_SYSCALL_64_fastpath+0x16/0x75 [Wed Mar 30 11:38:59 2016] Code: a2 d0 00 00 00 4c 89 a2 d8 00 00 00 c7 82 c8 00 00 00 00 00 00 00 44 89 82 cc 00 00 00 66 89 82 c6 00 00 00 66 89 8a c2 00 00 00 <48> c7 07 00 00 00 00 48 c7 47 08 00 00 00 00 48 c7 47 10 00 00 [Wed Mar 30 11:39:02 2016] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [exe:1315] [Wed Mar 30 11:39:02 2016] Modules linked in: btrfs(E) xor(E) raid6_pq(E) ufs(E) qnx4(E) hfsplus(E) hfs(E) minix(E) ntfs(E) msdos(E) jfs(E) xfs(E) libcrc32c(E) xt_nat xt_tcpudp veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables x_tables nf_nat nf_conntrack br_netfilter bridge stp llc aufs sit tunnel4 ip_tunnel i2c_piix4 input_leds serio_raw hyperv_fb hyperv_keyboard 8250_fintek joydev mac_hid autofs4 hid_generic hv_netvsc hid_hyperv hv_utils hid hv_storvsc crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cr
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Which ubuntu version are you using for tests? 14.04? How about vss daemon, kvp daemon? Have you installed them? Which version and package have you used? Are you using crash consistent or application consistent backups (with VSS support in the Guest Linux)? I'd like to independently repeat your tests to confirm the results yet I need to replicate your environment. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
No luck with 4.2.0-27 # uname -a Linux xxx 4.2.0-27-generic #32-Ubuntu SMP Fri Jan 22 04:49:08 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Just got this few days ago: [Tue Feb 9 00:00:56 2016] sd 2:0:0:0: [storvsc] Sense Key : Unit Attention [current] [Tue Feb 9 00:00:56 2016] sd 2:0:0:0: [storvsc] Add. Sense: Changed operating definition [Tue Feb 9 00:00:56 2016] sd 2:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automa [Tue Feb 9 00:01:36 2016] sd 2:0:0:0: [storvsc] Sense Key : Unit Attention [current] [Tue Feb 9 00:01:36 2016] sd 2:0:0:0: [storvsc] Add. Sense: Changed operating definition [Tue Feb 9 00:01:36 2016] sd 2:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automa [Tue Feb 9 00:01:36 2016] blk_update_request: I/O error, dev sda, sector 1075632 [Tue Feb 9 00:01:36 2016] Aborting journal on device sda1-8. [Tue Feb 9 00:01:36 2016] EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal [Tue Feb 9 00:01:36 2016] EXT4-fs (sda1): Remounting filesystem read-only [Wed Feb 10 00:02:49 2016] EXT4-fs (sda1): error count since last fsck: 2 [Wed Feb 10 00:02:49 2016] EXT4-fs (sda1): initial error at time 1454972496: ext4_journal_check_start:56 [Wed Feb 10 00:02:49 2016] EXT4-fs (sda1): last error at time 1454972496: ext4_journal_check_start:56 [Thu Feb 11 00:04:37 2016] EXT4-fs (sda1): error count since last fsck: 2 [Thu Feb 11 00:04:37 2016] EXT4-fs (sda1): initial error at time 1454972496: ext4_journal_check_start:56 [Thu Feb 11 00:04:37 2016] EXT4-fs (sda1): last error at time 1454972496: ext4_journal_check_start:56 [Fri Feb 12 00:06:24 2016] EXT4-fs (sda1): error count since last fsck: 2 [Fri Feb 12 00:06:24 2016] EXT4-fs (sda1): initial error at time 1454972496: ext4_journal_check_start:56 [Fri Feb 12 00:06:24 2016] EXT4-fs (sda1): last error at time 1454972496: ext4_journal_check_start:56 I ended with completely corrupted fs. fsck ended in all files in lost+found and garbled filenames. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: ht
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Actually I got this on 3 VMs on my Azure Pack. One 15.10 and two 14.04.03. On 15.10 I tried to fsck the root fs before reboot. What could go wrong as fs is already read-only I thought. Here is the session: ~# touch /oioi touch: cannot touch '/oioi': Read-only file system ~# fsck /<--- At this point it corrupted everything already. fsck from util-linux 2.26.2 e2fsck 1.42.12 (29-Aug-2014) cloudimg-rootfs: recovering journal fsck.ext4: Bad magic number in super-block while trying to re-open cloudimg-rootfs cloudimg-rootfs: ** WARNING: Filesystem still has errors ** ~# fsck -f / fsck from util-linux 2.26.2 fsck.ext4: Unable to resolve 'UUID=5370f2f3-e06a-4682-bc53-2f844792ebe1' ~# ls -la / total 0 After reboot grub said it cannot find FS so I booted it from another HDD and fcsked from that. It reported a lot of errors but rendered the FS as explained before - everything in lost+found. On 14.04 I rebooted without fsck. After boot it said that FS i corrupted so I can Skip mounting it (what root?) or try Manualy fix it. So I pressed the M but it looked like it's frozen. Luckily I restored all VMs from the backup. Yes, the very same backup that actually triggered the issue as we found it's 100% correlated that the corruption happens when the backup job starts on HV. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
@F-Bosch: Regarding the link going down I experienced that too. I correlated it with IPv6. When I disabled IPv6 on the interface the problem went away. Previously the more data I sent over ipv6 the sooner it broke. No logs whatsoever. Migrating the machine to another host restored the connectivity without rebooting the guest. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
I'm on the following since Feb 20: Linux XXX-Linux 4.2.0-30-generic #35-Ubuntu SMP Fri Feb 19 13:52:26 It still reports Add. Sense: [Wed Feb 24 00:01:01 2016] sd 2:0:0:0: [storvsc] Sense Key : Unit Attention [current] [Wed Feb 24 00:01:01 2016] sd 2:0:0:0: [storvsc] Add. Sense: Changed operating definition [Wed Feb 24 00:01:01 2016] sd 2:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automa [Wed Feb 24 00:01:31 2016] sd 2:0:0:0: [storvsc] Sense Key : Unit Attention [current] [Wed Feb 24 00:01:31 2016] sd 2:0:0:0: [storvsc] Add. Sense: Changed operating definition [Wed Feb 24 00:01:31 2016] sd 2:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automa On the other hand I experienced the very same crash on Debian with 4.3.0-0. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
@f-bosch: The message is an early warning. Even on the latest 4.2.0-30 it just forecasts the imminent failure: [Sat Feb 27 00:01:08 2016] sd 2:0:0:0: [storvsc] Sense Key : Unit Attention [current] [Sat Feb 27 00:01:08 2016] sd 2:0:0:0: [storvsc] Add. Sense: Changed operating definition [Sat Feb 27 00:01:08 2016] sd 2:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automa [Sat Feb 27 00:01:48 2016] sd 2:0:0:0: [storvsc] Sense Key : Unit Attention [current] [Sat Feb 27 00:01:48 2016] sd 2:0:0:0: [storvsc] Add. Sense: Changed operating definition [Sat Feb 27 00:01:48 2016] sd 2:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automa [Sat Feb 27 00:01:48 2016] blk_update_request: I/O error, dev sda, sector 1058512 [Sat Feb 27 00:01:48 2016] Aborting journal on device sda1-8. [Sat Feb 27 00:01:50 2016] EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal [Sat Feb 27 00:01:50 2016] EXT4-fs (sda1): Remounting filesystem read-only [Sun Feb 28 00:06:47 2016] EXT4-fs (sda1): error count since last fsck: 2 [Sun Feb 28 00:06:47 2016] EXT4-fs (sda1): initial error at time 1456527709: ext4_journal_check_start:56 [Sun Feb 28 00:06:47 2016] EXT4-fs (sda1): last error at time 1456527710: ext4_journal_check_start:56 [Mon Feb 29 00:08:35 2016] EXT4-fs (sda1): error count since last fsck: 2 [Mon Feb 29 00:08:35 2016] EXT4-fs (sda1): initial error at time 1456527709: ext4_journal_check_start:56 [Mon Feb 29 00:08:35 2016] EXT4-fs (sda1): last error at time 1456527710: ext4_journal_check_start:56 [Mon Feb 29 09:56:24 2016] sd 2:0:0:0: [storvsc] Sense Key : Unit Attention [current] [Mon Feb 29 09:56:24 2016] sd 2:0:0:0: [storvsc] Add. Sense: Changed operating definition [Mon Feb 29 09:56:24 2016] sd 2:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automa # uname -a Linux xxx-Linux 4.2.0-30-generic #35-Ubuntu SMP Fri Feb 19 13:52:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notificat
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
I experienced it during migration and other means involving snapshots. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Status in linux source package in Xenial: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Bad news. I tried to replicate the test results on my own. I downloaded and installed the first kernel marked as good: # uname -a Linux backup-01 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux I performed the tiobench and simultaneity the backup. At first it looked like the is no corruption as nothing worrying was reported in the dmesg. Unfortunately the OS behavior indicates that the there is at least memory corruption: root@backup-01:~# dpkg-reconfigure --frontend noninteractive tzdata Can't load '/usr/lib/perl/5.18/auto/IO/IO.so' for module IO: /usr/lib/perl/5.18/auto/IO/IO.so: failed to map segment from shared object: Cannot allocate memory at /usr/share/perl/5.18/XSLoader.pm line 68. at /usr/lib/perl/5.18/IO.pm line 11. Compilation failed in require at /usr/lib/perl/5.18/IO/Handle.pm line 9. BEGIN failed--compilation aborted at /usr/lib/perl/5.18/IO/Handle.pm line 9. Compilation failed in require at /usr/lib/perl/5.18/IO/Seekable.pm line 9. BEGIN failed--compilation aborted at /usr/lib/perl/5.18/IO/Seekable.pm line 9. Compilation failed in require at /usr/lib/perl/5.18/IO/File.pm line 11. BEGIN failed--compilation aborted at /usr/lib/perl/5.18/IO/File.pm line 11. Compilation failed in require at /usr/share/perl/5.18/FileHandle.pm line 9. Compilation failed in require at /usr/share/perl5/Debconf/Template.pm line 8. BEGIN failed--compilation aborted at /usr/share/perl5/Debconf/Template.pm line 8. Compilation failed in require at /usr/share/perl5/Debconf/Question.pm line 8. BEGIN failed--compilation aborted at /usr/share/perl5/Debconf/Question.pm line 8. Compilation failed in require at /usr/share/perl5/Debconf/Config.pm line 7. BEGIN failed--compilation aborted at /usr/share/perl5/Debconf/Config.pm line 7. Compilation failed in require at /usr/share/perl5/Debconf/Log.pm line 10. Compilation failed in require at /usr/share/perl5/Debconf/Db.pm line 7. BEGIN failed--compilation aborted at /usr/share/perl5/Debconf/Db.pm line 7. Compilation failed in require at /usr/sbin/dpkg-reconfigure line 11. BEGIN failed--compilation aborted at /usr/sbin/dpkg-reconfigure line 11. I observed some random segfaults as well: [ 7773.234696] tiotest[15052]: segfault at 0 ip 0040183e sp 7f2ed15a3f00 error 4 in tiotest[40+8000] [ 7774.185489] apt-get[15073]: segfault at ip 7f2568a3bbfb sp 7fff52a88840 error 5 in libapt-pkg.so.4.12.0[7f25689e5000+145000] After reboot it suddenly started to work: # dpkg-reconfigure --frontend noninteractive tzdata Current default time zone: 'SystemV/CST6CDT' Local time is now: Thu Apr 14 07:06:20 CDT 2016. Universal Time is now: Thu Apr 14 12:06:20 UTC 2016. That suggests that we're dealing with memory corruption bug that eventually leads to fs corruption. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Status in linux source package in Xenial: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" proc
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
What's your VSS snapshot frequency? Increasing it you can shorten the tests as the problem should appear earlier. I'm doing the snapshot every 15 minutes. Just snapshots, no need to do a full backup. In my case a snapshot made for any reason (branching, moving to different host etc.) was causing the issue with similar probability. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Status in linux source package in Xenial: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
I'm having difficulties installing those packages on trusty. I run into: linux-tools-3.13.0-86 : Depends: binutils (>= 2.24) but it is not going to be installed. Depends: binutils (< 2.25) but it is not going to be installed. Depends: libdw1 (>= 0.143) but it is not going to be installed. Depends: libunwind8 but it is not going to be installed. Depends: linux-tools-common but it is not going to be installed. BTW: by default in cloud image there are following packages installed: # dpkg -l | grep 3.19.0 ii linux-image-3.19.0-59-generic 3.19.0-59.66~14.04.1 amd64 Linux kernel image for version 3.19.0 on 64 bit x86 SMP ii linux-image-virtual-lts-vivid 3.19.0.59.42 amd64 This package will always depend on the latest minimal generic kernel image. ii linux-lts-vivid-cloud-tools-common 3.19.0-18.18~14.04.1 all Linux kernel version specific cloud tools for version 3.19.0 I'm wondering about linux-lts-vivid-cloud-tools-common... Do you provide the same version of cloud tools in your packages? (Note it's a vivid backport). -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Yakkety: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Testing... -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Yakkety: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Good news. # uname -a Linux backup-02 3.13.0-86-generic #131~lp1470250Commitd215f91Reverted SMP Wed May 25 20:47:20 UTC x86_64 x86_64 x86_64 GNU/Linux It is running since Jun 7 16:30:10 and still no crash. The only issue is that integration with VMM went awry and the hostname is reported as: getaddrinfo failed: 0xfffe Name or service not known. In the meantime I run 39.19.0-59 on the same cluster and it crashed twice (though vmm integration is OK). -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Yakkety: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Keep in mind tough that I suggested that it might be some kind of memory corruption bug and the filesystem corruption might be just manifestation. I experiences different problems on 3.13.0-32-generic (random crashes and so on) thet were related to snapshoting. I guess the vmm integration is made through hv_kvp_daemon so something is rotten in the state of Denmark. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Yakkety: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
http://orig00.deviantart.net/d4e7/f/2012/247/6/b/and_it_s_gone_by_celeith-d5div3y.jpg (sry, couldnt' resist it :) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Yakkety: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
It looks like it does the trick! Already over 90h of testing with tiobench and it works like a charm! Usually it took less than 4h (several at best) to crash. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Yakkety: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Yes. Linux backup-01 4.4.0-36-generic #55~lp1470250PatchedAlexFaulpeltz SMP Wed Aug 31 16:14:16 UTC 20 x86_64 x86_64 x86_64 GNU/Linux Another 24h and no crash. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Yakkety: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
I'm really looking forward to test the path on trusty. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Yakkety: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
The xenial kernel works like a charm since the patch. Great work! :) I'm using application consistent backup (the Hyper-V signals snapshot to the guest OS) rather than crash consistent (snapshoting without informing the guest). The Hyper-h integration services in the guest are required for that (I'm using ubuntu cloud images that are preloaded with that). I'll test the trusty patches now. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Yakkety: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Over 24h of trusty tests under heavy load and so good so far. IMHO the kernel patch should suffice. I don't see "trying to recover VSS connection" messages neither on 16.04 kernel nor 14.04. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Yakkety: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
The link to trusty kernels is gone :( http://kernel.ubuntu.com/~jsalisbury/lp1470250/AlexFaulpeltzPatch/trusty/ -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Yakkety: In Progress Status in linux source package in Zesty: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Are you aware of the fact that I have reported errors with 3.13.0-32-generic? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Yakkety: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
No luck with 14.04. After couple of days two test machines restarted unexpectedly at the very same moment. I sustepct kernel crash but the virtual console is gone and nothing was logged. On the other hand the xenial machine is working for weeks now without any issues. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Yakkety: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Thank you for the update. I'm staying tuned :) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Yakkety: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
