@ DuncanT and @dule: I noticed from IRC log that you are discussing about c-bak upgrade, and I am working on this, please see following message. Hope I don't miss anything.
As you know, currently c-bak and c-vol are in same nodes. c-bak depends on c-vol service. But it is not necessary that all c-vols needs to be upgraded before c-backs. The sequences can be random. As described in the patch https://review.openstack.org/#/c/269412/, when c-api decides which c-bak service to create/restore, it checks the version of c-vol. If c-vol is new version, find a c-bak in new version. If c-vol is in old version, find a c-bak in old version. Let's us think a real case. Customers upgrade c-api, c-sch, and start to upgrade c-vol and c-bak. There are two c-vol services c-vol1 and c-vol2, and two c-bak services c-bak1 and c-bak2. There are four typical upgrade sequences as following. Meanwhile, please notice that c-vol and c-bak are in same nodes in Liberty. So during upgrades, if c-vol went down to upgrade, c-bak is also down. 1. c-vol1->c-bak1->c-vol2->c-bak2 The only insufficiency is when c-vol1 upgrades, and other c-bak services haven't upgraded, the request to create/restore backups from/to volumes in c-vol1 will fail with reason "no valid c-bak service found". It is acceptable, as it is similar to scenario in Liberty: c-vol active and c-bak fails. 2. c-vol1->c-vol2->c-bak1->c-bak2 Before c-bak1 upgrades, no back request can be completed as no active c-bak services. This is reasonable. 3. c-bak1->c-vol1->c-bak2->c-vol2: The issue is when c-bak2 upgrades, the request to create/restore backups from/to volumes in c-vol2 will fail with reason c-vol not active. This is consistent with behaviors in Liberty. 4. c-bak1->c-bak2->c-vol1->c-vol2: Before c-vol1 upgrades, no back request can be completed as c-vol services not active This is reasonable. Best wishes Lisa On Jan 20, 2016 21:59, Michał Dulko wrote: > Hi, > > In Mitaka Cinder team is implementing rolling upgrades capabilities. I > want to get some feedback on possibilities of implementing some partial or > multinode grenade check for our purposes. > > Basically Cinder consists of 4 services calling each other over RPC the in > following fashion: > > +---------+ c-api +---------+ > | + | > | | | > v v v > c-sch <-----> c-vol <-----+ c-bak > > The order of upgrades we're forced to use (at least in this release) is > c-api->c-sch->c-vol->c-bak. I wonder how we can test interoperability of > services in different versions in that model. I have three ideas: > > 1. One idea would be to have two nodes - one with full Cinder deployment > in latest master and second one running c-sch, c-vol and c-bak in latest > stable version. That way we would test interoperability of the services > versions. A problem I see is that tests would be strongly nondeterministic, > as a particular test result would depend on which RPC service got the > request. That would make debugging CI failures harder and may result in > breaking patches slipping in. > > 2. We may simply run a controller<->compute multinode, similar to how > Nova is running multinode grenade job. Controller would run c-api, c-sch > and compute c-vol, c-bak. Disadvantage of this model is the fact that we > wouldn't test c-api->c-sch compatibility. > > 3. Do a single upgrade for each of the services. That would mean testing > master c-api with stable rest of services, then upgrading c-sch, retesting, > upgrading c-vol, retesting, upgrading c-bak, retesting. That way we would > test all the possibilities but such run would take a lot of time. Moreover if > I > recall correctly such idea isn't possible in current state of Grenade. > > Comments and feedback is very welcome. > > Thanks, > Michal (IRC: dulek) > _____________________________________________________________ _____________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: OpenStack-dev- requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Best wishes Lisa __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
