----- Original Message ----- > From: "Daniel P. Berrange" <berra...@redhat.com> > To: openstack-dev@lists.openstack.org, openstack-operat...@lists.openstack.org > Sent: Friday, 30 January, 2015 11:47:16 AM > Subject: [openstack-dev] [nova][libvirt] RFC: ensuring live migration ends > > In working on a recent Nova migration bug > > https://bugs.launchpad.net/nova/+bug/1414065 > > I had cause to refactor the way the nova libvirt driver monitors live > migration completion/failure/progress. This refactor has opened the > door for doing more intelligent active management of the live migration > process. > > As it stands today, we launch live migration, with a possible bandwidth > limit applied and just pray that it succeeds eventually. It might take > until the end of the universe and we'll happily wait that long. This is > pretty dumb really and I think we really ought to do better. The problem > is that I'm not really sure what "better" should mean, except for ensuring > it doesn't run forever. > > As a demo, I pushed a quick proof of concept showing how we could easily > just abort live migration after say 10 minutes > > https://review.openstack.org/#/c/151665/ > > There are a number of possible things to consider though... > > First how to detect when live migration isn't going to succeeed. > > - Could do a crude timeout, eg allow 10 minutes to succeeed or else. > > - Look at data transfer stats (memory transferred, memory remaining to > transfer, disk transferred, disk remaining to transfer) to determine > if it is making forward progress.
I think this is a better option. We could define a timeout for the progress and cancel if there is no progress. IIRC there were similar debates about it in Ovirt, we could do something similar: https://github.com/oVirt/vdsm/blob/master/vdsm/virt/migration.py#L430 > > - Leave it upto the admin / user to decided if it has gone long enough > > The first is easy, while the second is harder but probably more reliable > and useful for users. > > Second is a question of what todo when it looks to be failing > > - Cancel the migration - leave it running on source. Not good if the > admin is trying to evacuate a host. > > - Pause the VM - make it complete as non-live migration. Not good if > the guest workload doesn't like being paused > > - Increase the bandwidth permitted. There is a built-in rate limit in > QEMU overridable via nova.conf. Could argue that the admin should just > set their desired limit in nova.conf and be done with it, but perhaps > there's a case for increasing it in special circumstances. eg emergency > evacuate of host it is better to waste bandwidth & complete the job, > but for non-urgent scenarios better to limit bandwidth & accept failure ? > > - Increase the maximum downtime permitted. This is the small time window > when the guest switches from source to dest. To small and it'll never > switch, too large and it'll suffer unacceptable interuption. > In my opinion, it would be great if we could play with bandwidth and downtime before cancelling the migration or pausing. However, It makes sense only if there is some kind of a progress in the transfer stats and not a complete disconnect. In that case we should just cancel it. > We could do some of these things automatically based on some policy > or leave them upto the cloud admin/tenant user via new APIs > > Third there's question of other QEMU features we could make use of to > stop problems in the first place > > - Auto-converge flag - if you set this QEMU throttles back the CPUs > so the guest cannot dirty ram pages as quickly. This is nicer than > pausing CPUs altogether, but could still be an issue for guests > which have strong performance requirements > > - Page compression flag - if you set this QEMU does compression of > pages to reduce data that has to be sent. This is basically trading > off network bandwidth vs CPU burn. Probably a win unless you are > already highly overcomit on CPU on the host > > Fourth there's a question of whether we should give the tenant user or > cloud admin further APIs for influencing migration > > - Add an explicit API for cancelling migration ? > > - Add APIs for setting tunables like downtime, bandwidth on the fly ? > > - Or drive some of the tunables like downtime, bandwidth, or policies > like cancel vs paused from flavour or image metadata properties ? > > - Allow operations like evacuate to specify a live migration policy > eg switch non-live migrate after 5 minutes ? > IMHO, an explicit API for cancelling migration is very much needed. I remember cases when migrations took about 8 or hours, leaving the admins helpless :) Also, I very much like the idea of having tunables and policy to set in the flavours and image properties. To allow the administrators to set these as a "template" in the flavour and also to let the users to update/override or "request" these options as they should know the best (hopefully) what is running in their guests. > The current code is so crude and there's a hell of alot of options we > can take. I'm just not sure which is the best direction for us to go > in. > > What kind of things would be the biggest win from Operators' or tenants' > POV ? Thanks for writing this! :) > > Regards, > Daniel > -- > |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| > |: http://libvirt.org -o- http://virt-manager.org :| > |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| > |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
