[Bug 1006212] Re: cciss: hpacucli "ctrl slot=0 create type=ld drives=2:4" hangs, spews call trace in dmesg
Hello, As a data point, I'm seeing this same issue on 3 different DL360-G5's all running Precise. 2 systems are seeing the issue with the P400i controllers and the 3rd has the issue with a P800 controller and external disk arrays. I experienced the issue while creating RAID1 sets as well as 5 disk and 10 disk RAID5 sets. All are running with kernel 3.2.0-31-generic. Thanks, gary -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1006212 Title: cciss: hpacucli "ctrl slot=0 create type=ld drives=2:4" hangs, spews call trace in dmesg To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1006212/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 997978] Re: KVM images lose connectivity with bridged network
I don't believe just rebooting a guest will cause a new KVM instance to load. As a test, I just rebooted a guest VM on a system here and the pid of the kvm process did not change. I think it may be possible that you are still running on the old software. Also, to update my data point... On my server which was experiencing issues, I rebooted the host just to make sure everything was fresh. It's been about a month and I have not experienced the failure again. I was typically going a few weeks between issues. gary - Original Message - From: "Matt Hilt" To: g...@isgsoftware.net Sent: Tuesday, September 25, 2012 12:07:44 PM Subject: [Bug 997978] Re: KVM images lose connectivity with bridged network Soren, We have a 12.04 based OpenStack cluster with 4 host nodes running about 30 VMs currently. We performed the steps to add the kvm-network-hang repo and updated to the latest version on the host machines, then rebooted the instances. My understanding is that this should catch the update, since a new KVM command is run on reboot. I caught the first failure ~12 hours after the upgrade. It had the usual symptoms: networking loss, but the VM is still up and an active VNC session was possible. I thought I just might have missed a reboot on one of the VMs, so I didn't report anything. The second failure happened yesterday, but someone else caught it and rebooted the VM. As best we can tell after the fact, it looks like the usual failure (no full harddrive, or kernel panic, or anything that got logged). As I mentioned before, we used to see at least one failure per day, usually much more. This patch has at least reduced the occurence to a minimal amount. These non-deterministic bugs are hard to track down. -- You received this bug notification because you are subscribed to the bug report. https://bugs.launchpad.net/bugs/997978 Title: KVM images lose connectivity with bridged network Status in OpenStack Compute (Nova): Invalid Status in “qemu-kvm” package in Ubuntu: Fix Released Status in “qemu-kvm” source package in Precise: In Progress Bug description: = SRU Justification: 1. Impact: networking breaks after awhile in kvm guests using virtio networking 2. Development fix: The bug was fixed upstream and the fix picked up in a new merge. 3. Stable fix: 3 virtio patches are cherrypicked from upstream: a821ce5 virtio: order index/descriptor reads 92045d8 virtio: add missing mb() on enable notification a281ebc virtio: add missing mb() on notification 4. Test case: Create a bridge enslaving the real NIC, and use that as the bridge for a kvm instance with virtio networking. See comment #44 for specific test case. 5. Regression potential: Should be low as several people have tested the fixed package under heavy load. = System: --- Dell R410 Dual processor 2.4Ghz w/16G RAM Distributor ID: Ubuntu Description:Ubuntu 12.04 LTS Release:12.04 Codename: precise Setup: - We're running 3 KVM guests, all Ubuntu 12.04 LTS using bridged networking. From the host: # cat /etc/network/interfaces auto br0 iface br0 inet static address 212.XX.239.98 netmask 255.255.255.240 gateway 212.XX.239.97 bridge_ports eth0 bridge_fd 9 bridge_hello 2 bridge_maxage 12 bridge_stp off # ifconfig eth0 eth0 Link encap:Ethernet HWaddr d4:ae:52:84:2d:5a UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:11278363 errors:0 dropped:3128 overruns:0 frame:0 TX packets:14437384 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4115980743 (4.1 GB) TX bytes:5451961979 (5.4 GB) Interrupt:36 Memory:da00-da012800 # ifconfig br0 br0 Link encap:Ethernet HWaddr d4:ae:52:84:2d:5a inet addr:212.XX.239.98 Bcast:212.XX.239.111 Mask:255.255.255.240 inet6 addr: fe80::d6ae:52ff:fe84:2d5a/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1720861 errors:0 dropped:0 overruns:0 frame:0 TX packets:1708622 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:210152198 (210.1 MB) TX bytes:300858508 (300.8 MB) # brctl show bridge name bridge id STP enabled interfaces br0 8000.d4ae52842d5a no eth0 I have no default network configured to autostart in libvirt as we're using bridged networking: # virsh net-list --all Name State Autostart - default inactive no # arp Address HWtype HWaddress Flags Mask Iface mailer03..co
[Bug 997978] Re: KVM images lose connectivity with bridged network
I have seen this issue on 2 different servers which use bridging but not bonding. One server was a customer system and we were forced to back-date the OS to an earlier release. They were experiencing the issue up to once/day and quickly got impatient to have it resolved. The other server is an internal system which runs multiple vm's. We have only seen the issue on one of the vm's and only once every 2-3 weeks. The vm which experiences the issue is our LTSP server. I have been testing a small cluster of 3 host machines which use both bonding and bridging. I have not seen this issue affect them, but the usage is quite light and the vm's come & go since it's a testing environment right now. Due to this bug, we have halted any plans to upgrade vm hosts to Precise until we can verify it's fixed. We've seen the following when the issue has occurred: * Absolutely nothing in any logs, dmesg, etc. * Host machine cannot ping the guest * arp shows guest as incomplete * guest machine can ping its own IP, but nothing else (host, gw, etc) * restarting networking subsystem is successful (no errors) but has no effect on the problem * rebooting the guest fixes the problem until it happens again. The reboot does not actually kill the kvm session and get a new process ID, but somehow having the guest go through the init again fixes it (until it happens again some period later). * This issue has occurred on one 12.04 guest and one 11.10 guest * Both of the servers which this occured on are Dell 2950 series machines. I have not seen this issue on any of our HP Proliant (mostly DL360's) machines. If there is some sort of test I can run to help debug, I'm happy to do that. Thank you for trying to address this. This is a huge bug for us. Thanks, gary -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/997978 Title: KVM images lose connectivity with bridged network To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/997978/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 997978] Re: KVM images lose connectivity with bridged network
Regarding comment #55: I don't believe this is a guest OS issue. In my comment #51 I also indicate that I have had non-precise vm's (specifically 11.10) experience the same issue. I have only experienced this issue when using 12.04 precise as the host OS. On a customer server which I back-dated by reinstalling 11.10 as the host OS, the issue went away. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/997978 Title: KVM images lose connectivity with bridged network To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/997978/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 997978] Re: KVM images lose connectivity with bridged network
Hi Serge, I would be willing to install your ppa packages. The only issue on my end is that this problem only occurs every 3-4 weeks for me. So I don't think I would be able to give any sort of concrete feedback for whether it addresses the issue or not. My customer's server was having the issue several times per week, but I had to take action pretty quickly to keep them happy and chose to backdate to 11.10. After 9 days, they have not had the issue, which lends more evidence that it is not a guest-based issue but host-based. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/997978 Title: KVM images lose connectivity with bridged network To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/997978/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 997978] Re: KVM images lose connectivity with bridged network
I installed the new packages on Monday and have not had any issues. That said, I was only experiencing failures once every 3-4 weeks. So I don't think my data point will be valid for at least a few weeks. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/997978 Title: KVM images lose connectivity with bridged network To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/997978/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1006212] Re: cciss: hpacucli "ctrl slot=0 create type=ld drives=2:4" hangs, spews call trace in dmesg
One of my systems which experiences the problem is running hpacucli v8.70-8.0. The system is 64-bit. I'm running the utility using the following command: setarch x86_64 --uname-2.6 hpacucli Without the setarch, the utility does not detect any controllers. Hope the information helps, gary -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1006212 Title: cciss: hpacucli "ctrl slot=0 create type=ld drives=2:4" hangs, spews call trace in dmesg To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1006212/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
