Re: [openstack-dev] [nova] top gate bug is libvirt snapshot

2014-07-15 Thread Alex Xu
Question about swap volume, swap volume's implementation is very similar with live snapshot. Both implemented by blockRebase. But swap volume didn't check any libvirt and qemu version. Should we add version check for swap_volume now? That means swap_volume will be disable also. On 2014?06?26?

Re: [openstack-dev] [nova] top gate bug is libvirt snapshot

2014-07-10 Thread Daniel P. Berrange
On Thu, Jul 10, 2014 at 08:32:37AM -0400, Sean Dague wrote: > On 07/10/2014 05:03 AM, Daniel P. Berrange wrote: > > On Wed, Jul 09, 2014 at 06:23:27PM -0400, Sean Dague wrote: > >> The libvirt logs needed are huge, so we can't run them all the time. And > >> realistically, I don't think they provid

Re: [openstack-dev] [nova] top gate bug is libvirt snapshot

2014-07-10 Thread Sean Dague
On 07/10/2014 05:03 AM, Daniel P. Berrange wrote: > On Wed, Jul 09, 2014 at 06:23:27PM -0400, Sean Dague wrote: >> The libvirt logs needed are huge, so we can't run them all the time. And >> realistically, I don't think they provided us the info we needed. There >> has been at least one fail on Dan

Re: [openstack-dev] [nova] top gate bug is libvirt snapshot

2014-07-10 Thread Daniel P. Berrange
On Wed, Jul 09, 2014 at 06:23:27PM -0400, Sean Dague wrote: > The libvirt logs needed are huge, so we can't run them all the time. And > realistically, I don't think they provided us the info we needed. There > has been at least one fail on Dan's log hack patch for this scenario > today, so maybe i

Re: [openstack-dev] [nova] top gate bug is libvirt snapshot

2014-07-09 Thread Sean Dague
The libvirt logs needed are huge, so we can't run them all the time. And realistically, I don't think they provided us the info we needed. There has been at least one fail on Dan's log hack patch for this scenario today, so maybe it will be in there. On 07/09/2014 05:44 PM, Michael Still wrote: >

Re: [openstack-dev] [nova] top gate bug is libvirt snapshot

2014-07-09 Thread Michael Still
Can we get our gate images tweaked to have more verbose libvirt logging on in general? There's been a few times in the last year or so when we've really needed it. Michael On Wed, Jul 9, 2014 at 6:01 PM, Daniel P. Berrange wrote: > On Wed, Jul 09, 2014 at 08:58:02AM +1000, Michael Still wrote: >

Re: [openstack-dev] [nova] top gate bug is libvirt snapshot

2014-07-09 Thread Kashyap Chamarthy
On Wed, Jul 09, 2014 at 05:47:47PM +0530, Kashyap Chamarthy wrote: > On Tue, Jul 08, 2014 at 06:21:31PM -0400, Sean Dague wrote: > > On 07/08/2014 06:12 PM, Joe Gordon wrote: > > > > > > > > > > > > On Tue, Jul 8, 2014 at 2:56 PM, Michael Still > > > wrote: > > > > >

Re: [openstack-dev] [nova] top gate bug is libvirt snapshot

2014-07-09 Thread Daniel P. Berrange
On Wed, Jul 09, 2014 at 08:34:06AM -0400, Sean Dague wrote: > On 07/09/2014 03:58 AM, Daniel P. Berrange wrote: > > On Tue, Jul 08, 2014 at 02:50:40PM -0700, Joe Gordon wrote: > But for right now, we should stop the bleeding, so that nova/libvirt > isn't blocking everyone else from mergin

Re: [openstack-dev] [nova] top gate bug is libvirt snapshot

2014-07-09 Thread Sean Dague
On 07/09/2014 03:58 AM, Daniel P. Berrange wrote: > On Tue, Jul 08, 2014 at 02:50:40PM -0700, Joe Gordon wrote: But for right now, we should stop the bleeding, so that nova/libvirt isn't blocking everyone else from merging code. >>> >>> Agreed, we should merge the hack and treat the bug a

Re: [openstack-dev] [nova] top gate bug is libvirt snapshot

2014-07-09 Thread Kashyap Chamarthy
On Tue, Jul 08, 2014 at 06:21:31PM -0400, Sean Dague wrote: > On 07/08/2014 06:12 PM, Joe Gordon wrote: > > > > > > > > On Tue, Jul 8, 2014 at 2:56 PM, Michael Still > > wrote: > > > > The associated bug says this is probably a qemu bug, so I think we > > shou

Re: [openstack-dev] [nova] top gate bug is libvirt snapshot

2014-07-09 Thread Daniel P. Berrange
On Wed, Jul 09, 2014 at 08:58:02AM +1000, Michael Still wrote: > On Wed, Jul 9, 2014 at 8:21 AM, Sean Dague wrote: > > > This is also why I find it unlikely to be a qemu bug, because that's not > > shared state between guests. If qemu just randomly wedges itself, that > > would be detectable much

Re: [openstack-dev] [nova] top gate bug is libvirt snapshot

2014-07-09 Thread Daniel P. Berrange
On Tue, Jul 08, 2014 at 02:50:40PM -0700, Joe Gordon wrote: > > > But for right now, we should stop the bleeding, so that nova/libvirt > > > isn't blocking everyone else from merging code. > > > > Agreed, we should merge the hack and treat the bug as release blocker > > to be resolve prior to Juno

Re: [openstack-dev] [nova] top gate bug is libvirt snapshot

2014-07-08 Thread Michael Still
On Wed, Jul 9, 2014 at 8:21 AM, Sean Dague wrote: > This is also why I find it unlikely to be a qemu bug, because that's not > shared state between guests. If qemu just randomly wedges itself, that > would be detectable much easier outside of the gate. And there have been > attempts by danpb to s

Re: [openstack-dev] [nova] top gate bug is libvirt snapshot

2014-07-08 Thread Sean Dague
On 07/08/2014 06:12 PM, Joe Gordon wrote: > > > > On Tue, Jul 8, 2014 at 2:56 PM, Michael Still > wrote: > > The associated bug says this is probably a qemu bug, so I think we > should rephrase that to "we need to start thinking about how to make > sure up

Re: [openstack-dev] [nova] top gate bug is libvirt snapshot

2014-07-08 Thread Joe Gordon
On Tue, Jul 8, 2014 at 2:56 PM, Michael Still wrote: > The associated bug says this is probably a qemu bug, so I think we > should rephrase that to "we need to start thinking about how to make > sure upstream changes don't break nova". > Good point. Would running devstack-tempest on the latest

Re: [openstack-dev] [nova] top gate bug is libvirt snapshot

2014-07-08 Thread Boris Pavlovic
Joe, What about running benchmarks (with small load), for all major functions (like snaphshoting, booting/deleting, ..) on every patch in nova. It can catch a lot of related stuff. Best regards, Boris Pavlovic On Wed, Jul 9, 2014 at 1:56 AM, Michael Still wrote: > The associated bug says thi

Re: [openstack-dev] [nova] top gate bug is libvirt snapshot

2014-07-08 Thread Michael Still
The associated bug says this is probably a qemu bug, so I think we should rephrase that to "we need to start thinking about how to make sure upstream changes don't break nova". Michael On Wed, Jul 9, 2014 at 7:50 AM, Joe Gordon wrote: > > On Thu, Jun 26, 2014 at 4:12 AM, Daniel P. Berrange > wr

Re: [openstack-dev] [nova] top gate bug is libvirt snapshot

2014-07-08 Thread Joe Gordon
On Thu, Jun 26, 2014 at 4:12 AM, Daniel P. Berrange wrote: > On Thu, Jun 26, 2014 at 07:00:32AM -0400, Sean Dague wrote: > > While the Trusty transition was mostly uneventful, it has exposed a > > particular issue in libvirt, which is generating ~ 25% failure rate now > > on most tempest jobs. >

Re: [openstack-dev] [nova] top gate bug is libvirt snapshot

2014-06-26 Thread Daniel P. Berrange
On Thu, Jun 26, 2014 at 07:00:32AM -0400, Sean Dague wrote: > While the Trusty transition was mostly uneventful, it has exposed a > particular issue in libvirt, which is generating ~ 25% failure rate now > on most tempest jobs. > > As can be seen here - > https://github.com/openstack/nova/blob/mas

[openstack-dev] [nova] top gate bug is libvirt snapshot

2014-06-26 Thread Sean Dague
While the Trusty transition was mostly uneventful, it has exposed a particular issue in libvirt, which is generating ~ 25% failure rate now on most tempest jobs. As can be seen here - https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L294-L297 ... the libvirt live_snapsho