On Wed, Jun 24, 2015 at 11:02 AM, Joe Gordon <joe.gord...@gmail.com> wrote:
> > > On Wed, Jun 24, 2015 at 11:01 AM, Joe Gordon <joe.gord...@gmail.com> > wrote: > >> >> >> On Wed, Jun 24, 2015 at 10:45 AM, Sean Dague <s...@dague.net> wrote: >> >>> On 06/24/2015 01:31 PM, Joe Gordon wrote: >>> > >>> > >>> > On Tue, Jun 16, 2015 at 9:58 AM, Sean Dague <s...@dague.net >>> > <mailto:s...@dague.net>> wrote: >>> > >>> > Back when Nova first wanted to test partial upgrade, we did a >>> bunch of >>> > slightly odd conditionals inside of grenade and devstack to make >>> it so >>> > that if you were very careful, you could just not stop some of the >>> old >>> > services on a single node, upgrade everything else, and as long as >>> the >>> > old services didn't stop, they'd be running cached code in memory, >>> and >>> > it would look a bit like a 2 node worker not upgraded model. It >>> worked, >>> > but it was weird. >>> > >>> > There has been some interest by the Nova team to expand what's not >>> being >>> > touched, as well as the Neutron team to add partial upgrade testing >>> > support. Both are great initiatives, but I think going about it >>> the old >>> > way is going to add a lot of complexity in weird places, and not >>> be as >>> > good of a test as we really want. >>> > >>> > Nodepool now supports allocating multiple nodes. We have a >>> multinode job >>> > in Nova regularly testing live migration using this. >>> > >>> > If we slice this problem differently, I think we get a better >>> > architecture, a much easier way to add new configs, and a much more >>> > realistic end test. >>> > >>> > Conceptually, use devstack-gate multinode support to set up 2 >>> nodes, an >>> > all in one, and a worker. Let grenade upgrade the all in one, >>> leave the >>> > worker alone. >>> > >>> > I think the only complexity here is the fact that grenade.sh >>> implicitly >>> > drives stack.sh. Which means one of: >>> > >>> > 1) devstack-gate could build the worker first, then run grenade.sh >>> > >>> > 2) we make it so grenade.sh can execute in parts more easily, so >>> it can >>> > hand something else running stack.sh for it.' >>> > >>> > 3) we make grenade understand the subnode for partial upgrade, so >>> it >>> > will run the stack phase on the subnode itself (given credentials). >>> > >>> > This kind of approach means deciding which services you don't want >>> to >>> > upgrade doesn't require devstack changes, it's just a change of the >>> > services on the worker. >>> > >>> > We need a volunteer for taking this on, but I think all the follow >>> on >>> > partial upgrade support will be much much easier to do after we >>> have >>> > this kind of mechanism in place. >>> > >>> > >>> > I think this is a great approach for the future of partial upgrade >>> > support in grenade. I would like to point out step 0 here, is to get >>> > tempest passing consistently in multinode. >>> > >>> > Currently the neutron job is failing consistently, and nova-network >>> > fails roughly 10% of the time due >>> > to https://bugs.launchpad.net/nova/+bug/1462305 >>> > and https://bugs.launchpad.net/nova/+bug/1445569 >>> >>> Grenade is only running tempest smoke, which is a quite small number of >>> tests (and not the shelve/unshelve one for instance). I would expect >>> it's pass rate to be much higher. >>> >>> >> One way to find out. Want to get a multinode tempest smoke job running >> and see how it looks after running for a few days. >> > > smoke jobs*, one for nova-net and one for neutron. > > Proposal for multinode smoke jobs: https://review.openstack.org/#/c/195259/ > >> >>> -Sean >>> >>> -- >>> Sean Dague >>> http://dague.net >>> >>> >>> __________________________________________________________________________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
