+Billy McFall
On 11/17/2016 04:49 AM, Yuanhan Liu wrote: > On Thu, Nov 17, 2016 at 09:47:09AM +0100, Maxime Coquelin wrote: >> >> On 11/17/2016 09:29 AM, Yuanhan Liu wrote: >>> As usaual, sorry for late response :/ >>> >>> On Thu, Oct 13, 2016 at 08:50:52PM +0300, Michael S. Tsirkin wrote: >>>> Hi! >>>> So it looks like we face a problem with cross-version >>>> migration when using vhost. It's not new but became more >>>> acute with the advent of vhost user. >>>> >>>> For users to be able to migrate between different versions >>>> of the hypervisor the interface exposed to guests >>>> by hypervisor must stay unchanged. >>>> >>>> The problem is that a qemu device is connected >>>> to a backend in another process, so the interface >>>> exposed to guests depends on the capabilities of that >>>> process. >>>> >>>> Specifically, for vhost user interface based on virtio, this includes >>>> the "host features" bitmap that defines the interface, as well as more >>>> host values such as the max ring size. Adding new features/changing >>>> values to this interface is required to make progress, but on the other >>>> hand we need ability to get the old host features to be compatible. >>> It looks like to the same issue of vhost-user reconnect to me. For example, >>> >>> - start dpdk 16.07 & qemu 2.5 >>> - kill dpdk >>> - start dpdk 16.11 >>> >>> Though DPDK 16.11 has more features comparing to dpdk 16.07 (say, indirect), >>> above should work. Because qemu saves the negotiated features before the >>> disconnect and stores it back after the reconnection. >>> >>> commit a463215b087c41d7ca94e51aa347cde523831873 >>> Author: Marc-Andr? Lureau <marcandre.lureau at redhat.com> >>> Date: Mon Jun 6 18:45:05 2016 +0200 >>> >>> vhost-net: save & restore vhost-user acked features >>> >>> The initial vhost-user connection sets the features to be negotiated >>> with the driver. Renegotiation isn't possible without device reset. >>> >>> To handle reconnection of vhost-user backend, ensure the same set of >>> features are provided, and reuse already acked features. >>> >>> Signed-off-by: Marc-Andr? Lureau <marcandre.lureau at redhat.com> >>> >>> >>> So we could do similar to vhost-user? I mean, save the acked features >>> before migration and store it back after it. This should be able to >>> keep the compatibility. If user downgrades DPDK version, it also could >>> be easily detected, and then exit with an error to user: migration >>> failed due to un-compatible vhost features. >>> >>> Just some rough thoughts. Makes tiny sense? >> My understanding is that the management tool has to know whether >> versions are compatible before initiating the migration: > Makes sense. How about getting and restoring the acked features through > qemu command lines then, say, through the monitor interface? > > With that, it would be something like: > > - start vhost-user backend (DPDK, VPP, or whatever) & qemu in the src host > > - read the acked features (through monitor interface) > > - start vhost-user backend in the dst host > > - start qemu in the dst host with the just queried acked features > > QEMU then is expected to use this feature set for the later vhost-user > feature negotitation. Exit if features compatibility is broken. > > Thoughts? > > --yliu > >> 1. The downtime could be unpredictable if a VM has to move from hosts >> to hosts multiple times, which is problematic, especially for NFV. >> 2. If migration is not possible, maybe the management tool would >> prefer not to interrupt the VM on current host. >> >> I have little experience with migration though, so I could be mistaken. >> >> Thanks, >> Maxime >> >>> --yliu >>>> To solve this problem within qemu, qemu has a versioning system based on >>>> a machine type concept which fundamentally is a version string, by >>>> specifying that string one can get hardware compatible with a previous >>>> qemu version. QEMU also reports the latest version and list of versions >>>> supported so libvirt records the version at VM creation and then is >>>> careful to use this machine version whenever it migrates a VM. >>>> >>>> One might wonder how is this solved with a kernel vhost backend. The >>>> answer is that it mostly isn't - instead an assumption is made, that >>>> qemu versions are deployed together with the kernel - this is generally >>>> true for downstreams. Thus whenever qemu gains a new feature, it is >>>> already supported by the kernel as well. However, if one attempts >>>> migration with a new qemu from a system with a new to old kernel, one >>>> would get a failure. >>>> >>>> In the world where we have multiple userspace backends, with some of >>>> these supplied by ISVs, this seems non-realistic. >>>> >>>> IMO we need to support vhost backend versioning, ideally >>>> in a way that will also work for vhost kernel backends. >>>> >>>> So I'd like to get some input from both backend and management >>>> developers on what a good solution would look like. >>>> >>>> If we want to emulate the qemu solution, this involves adding the >>>> concept of interface versions to dpdk. For example, dpdk could supply a >>>> file (or utility printing?) with list of versions: latest and versions >>>> supported. libvirt could read that and >>>> - store latest version at vm creation >>>> - pass it around with the vm >>>> - pass it to qemu >>>> >>>> >From here, qemu could pass this over the vhost-user channel, >>>> thus making sure it's initialized with the correct >>>> compatible interface. >>>> >>>> As version here is an opaque string for libvirt and qemu, >>>> anything can be used - but I suggest either a list >>>> of values defining the interface, e.g. >>>> any_layout=on,max_ring=256 >>>> or a version including the name and vendor of the backend, >>>> e.g. "org.dpdk.v4.5.6". >>>> >>>> Note that typically the list of supported versions can only be >>>> extended, not shrunk. Also, if the host/guest interface >>>> does not change, don't change the current version as >>>> this just creates work for everyone. >>>> >>>> Thoughts? Would this work well for management? dpdk? vpp? >>>> >>>> Thanks! >>>> >>>> -- >>>> MST > _______________________________________________ > vpp-dev mailing list > vpp-dev at lists.fd.io > https://lists.fd.io/mailman/listinfo/vpp-dev -- *Thomas F Herbert* SDN Group Office of Technology *Red Hat*
