* Stefan Hajnoczi (stefa...@redhat.com) wrote: > On Tue, Nov 03, 2020 at 12:17:09PM +0000, Dr. David Alan Gilbert wrote: > > * Stefan Hajnoczi (stefa...@redhat.com) wrote: > > > Device Models > > > ------------- > > > Devices have a *hardware interface* consisting of hardware registers, > > > interrupts, and so on. > > > > > > The hardware interface together with the device state representation is > > > called > > > a *device model*. Device models can be assigned URIs such as > > > https://qemu.org/devices/e1000e to uniquely identify them. > > > > I think this is a unique identifier, not actually a URI; the https:// > > isn't needed since no one expects to ever connect to this. > > Yes, it could be any unique string. If the URI idea is not popular we > can use any similar scheme.
I'm OK with it being a URI; just drop the https. > > > However, secondary aspects related to the physical port may affect the > > > device's > > > hardware interface and need to be reflected in the device configuration. > > > The > > > link speed may depend on the physical port and be reported through the > > > device's > > > hardware interface. In that case a ``link-speed`` configuration parameter > > > is > > > required to prevent unexpected changes to the link speed after migration. > > > > That's an interesting example; because depending on the device, it might > > be: > > a) Completely virtualised so that the guest *shouldn't* know what > > the physical link speed is, precisely to allow the physical network on > > the destination to be different. > > > > b) Part of the migrated state > > > > c) Something that's allowed to be reloaded after migration > > > > d) Configurable > > > > so I'm not sure whether it's a good example in this case or not. > > Can you think of an example that has only one option? > > I tried but couldn't. For example take a sound card. The guest is aware > the device supports stereo playback (2 output channels), but which exact > stereo host device is used doesn't matter, they are all suitable. > > Now imagine migrating to a 7.1 surround-sound device. Similar options > come into play: > > a) Emulate stereo and mix it to 7.1 surround-sound on the physical > device. The guest still sees the stereo device. > > b) Refuse migration. > > c) Indicate that the output has switched and let the guest reconfigure > itself (e.g. a sound card with multiple outputs, where one of them is > stereo and another is 7.1 surround sound). > > Which option is desirable depends on the use case. Yes, but I think it might be worth calling out these differences; there are explicitly cases where you don't want external changes to be visible and other cases where you do; both are valid, but both need thinking about. (Another one, GPU whether you have a monitor plugged in!) > > Maybe what's needed is a stronger instruction to abstract external > > device state so that it's not part of the configuration in most cases. > > Do you want to propose something? I think something like 'Some part of a devices state may be irrelevant to a migration; for example on some NICs it might be preferable to hide the physical characteristics of the link from the guest.' > > > For example, if address filtering support was added to a network card then > > > device versions and the corresponding configurations may look like this: > > > * ``version=1`` - Behaves as if ``rx-filter-size=0`` > > > * ``version=2`` - ``rx-filter-size=32`` > > > > Note configuration parameters might have been added during the life of > > the device; e.g. if the original card had no support for rx-filters, it > > might not have a rx-filter-size parameter. > > version=1 does not explicitly set rx-filter-size=0. When a new parameter > is introduced it must have a default value that disables its effect on > the hardware interface and/or device state representation. This is > described in a bit more detail in the next section, maybe it should be > reordered. We've generally found the definition of devices tends in practice to be done newer->older; i.e. you define the current machine, and then define the next older machine setting the defaults that used to be true; then define the older version behind that.... > > > Device States > > > ------------- > > > The details of the device state representation are not covered in this > > > document > > > but the general requirements are discussed here. > > > > > > The device state consists of data accessible through the device's hardware > > > interface and internal state that is needed to restore device operation. > > > State in the hardware interface includes the values of hardware registers. > > > An example of internal state is an index value needed to avoid processing > > > queued requests more than once. > > > > I try and emphasise that 'internal state' should be represented in a way > > that reflects the problem rather than the particular implementation; > > this gives it a better chance of migrating to future versions. > > Sounds like a good idea. > > > > Changes can be made to the device state representation as follows. Each > > > change > > > to device state must have a corresponding device configuration parameter > > > that > > > allows the change to toggled: > > > > > > * When the parameter is disabled the hardware interface and device state > > > representation are unchanged. This allows old device states to be > > > loaded. > > > > > > * When the parameter is enabled the change comes into effect. > > > > > > * The parameter's default value disables the change. Therefore old > > > versions do > > > not have to explicitly specify the parameter. > > > > > > The following example illustrates migration from an old device > > > implementation to a new one. A version=1 network card is migrated to a > > > new device implementation that is also capable of version=2 and adds the > > > rx-filter-size=32 parameter. The new device is instantiated with > > > version=1, which disables rx-filter-size and is capable of loading the > > > version=1 device state. The migration completes successfully but note > > > the device is still operating at version=1 level in the new device. > > > > > > The following example illustrates migration from a new device > > > implementation back to an older one. The new device implementation > > > supports version=1 and version=2. The old device implementation supports > > > version=1 only. Therefore the device can only be migrated when > > > instantiated with version=1 or the equivalent full configuration > > > parameters. > > > > I'm sometimes asked for 'ways out' of buggy migration cases; e.g. what > > happens if version=1 forgot to migrate the X register; or what happens > > if verison=1 forgot to handle the special, rare case when X=5 and we > > now need to migrate some extra state. > > Can these cases be handled by adding additional configuration parameters? > > If version=1 is lacks essential state then version=2 can add it. The > user must configure the device to use version before they can save the > full state. > > If version=1 didn't handle the X=5 case then the same solution is > needed. A new configuration parameter is introduced and the user needs > to configure the device to be the new version before migrating. > > Unfortunately this requires poweroff or hotplugging a new device > instance. But some disruption is probably necessarily anyway so the > migration code on the host side can be patched to use the updated device > state representation. There are some corner cases that people sometimes prefer; for example lets say the X=5 case is actually really rare - but when it happens the device is hopelessly broken, some device authors prefer to fix it and send the extra data and let the migration fail if the destination doesn't understand it (it would break anyway). I've also been asked by mst for a 'unexpected data' mechanism to send data that the destination might not expect if it didn't know about it, for similar cases. > > > Orchestrating Migrations > > > ------------------------ > > > The following steps must be followed to migrate devices: > > > > > > 1. Check that the source and destination devices support the same device > > > model. > > > > > > 2. Check that the destination device supports the source device's > > > configuration. Each configuration parameter must be accepted by the > > > destination in order to ensure that it will be possible to load the > > > device > > > state. > > > > This is written in terms of a 'check'; there are at least three tricky > > things: > > > > a) Where they both have the same parameter, do they accept the same > > range of values; e.g. a newer version of the card might allow > > rx-filter-size to go upto 128 > > The easy way to handle that without lots of metadata is by instantiating > the destination device to see if it works. > > But in the next point you mention cloud where we need a way to find a > host that supports a given device. Metadata is probably needed to make > that check easy. In the email reply to Daniel Berrange I posted the > beginning of a JSON schema that describes device models for this > purpose. I think that offers a solution for the cloud case. A similar suggestion had come up in the vfio thread with Nvidia some months ago; I can't remember the outcome of that. (Much of this thread repeats the repeated long discussions on that thread!) Dave > > Stefan -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
