On 12/07/2015 04:28 AM, Michał Jastrzębski wrote:
On 6 December 2015 at 09:33, Jay Pipes <jaypi...@gmail.com> wrote:

On 12/04/2015 11:50 PM, Michał Jastrzębski wrote:

Hey guys,

Orchiestrated upgrades is one of our highest priorities for M in
kolla, so following up after discussion on summit I'd like to
suggest an approach:

Instead of creating playbook called "upgrade my openstack" we
will create "upgrade my nova" instead and approach to each
service case by case (since all of our running services are in
dockers, this is possible). We will also make use of image tags
as version carriers, so ansible will deploy new container only if
version tag differs from what we ask it to deploy. This will help
with idempotency of upgrade process.

So let's start with nova. Upgrade-my-nova playbook will do
something like this:

0. We create snapshot of our mariadb-data container. This will
affect every service, but it's always good to have backup and
rollback of db will be manual action

1. Nova bootstrap will be called and it will perform
db-migration. Since current approach to nova code is add-only we
shouldn't need to stop services and old services should keep
working on newer database. Also for minor version upgrades there
will be no action here unless there is migration. 2. We upgrade
all conductor at the same time. This should take mere seconds
since we'll have prebuilt containers 3. We will upgrade rest of
controller services with using "series: 1" in ansible to ensure
rolling upgrades. 4. We will upgrade all of nova-compute services
on it's own pace.

This workflow should be pretty robust (I think it is) and it
should also provide idempotency.


From Nova's perspective, most of the above looks OK to me, however,
I am in more in favor of an upgrade strategy that builds a "new"
set of conductor or controller service containers all at once and
then flips the VIP or managed IP from the current controller pod
containers to the newly-updated controller pod containers.

Well, we can't really do that because that would affect all other
services on controller host, and we want to minimize impact on rest
of cluster during upgrade. Containers gives us option to upgrade
"just nova", or even "just nova conductors" without affecting
anything else. On the bright side, redeploying pre-built container is
a matter of seconds (or even less). Whole process from turning off
last conductor to spawn first new conductor should take less than
10s, and that's only downtime we should get there.

Sorry, I guess I wasn't clear. I was not proposing to put all controller services in a single container. I was proposing simply to build the various container sets (nova-conductor, nova-api, etc) with the updated code for that service and then "flip the target IP or port" for that particular service from the existing container set's VIP to the new/updated container set's VIP.

In Kubernetes-speak, you would create a new Pod with containers having the updated Nova conductor code, and would set the new Pod's selector to something different than the existing Pod's selector -- using the Git SHA1 hash for the selector would be perfect... You would then update the Service object's selector to target the new Pod instead of the old one.

Or, alternately, you could use Kubernetes' support for rolling updates [1] of a service using a ReplicationController that essentially does the Pod and Service orchestration for you.

Hope that is a little more clear what I was referring to...

Best,
-jay

[1] https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/replication-controller.md#rolling-updates

This seems to me to be the most efficient (in terms of minimal
overall downtime) and most robust (because a "rollback" is simply
flipping the VIP back to the original controller/conductor service
pod containers) solution for upgrading Nova.



Dan Smith, can you comment on this since you are the God of
Upgrades?

Best, -jay


__________________________________________________________________________


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to