On Wed, 16 Oct 2019 22:18:47 +0200 Jens Freimann <jfreim...@redhat.com> wrote:
> On Tue, Oct 15, 2019 at 07:52:12PM -0600, Alex Williamson wrote: > >On Fri, 11 Oct 2019 13:20:15 +0200 > >Jens Freimann <jfreim...@redhat.com> wrote: > > > >> As usual block all vfio-pci devices from being migrated, but make an > >> exception for failover primary devices. This is achieved by setting > >> unmigratable to 0 but also add a migration blocker for all vfio-pci > >> devices except failover primary devices. These will be unplugged before > >> migration happens by the migration handler of the corresponding > >> virtio-net standby device. > >> > >> Signed-off-by: Jens Freimann <jfreim...@redhat.com> > >> --- > >> hw/vfio/pci.c | 35 ++++++++++++++++++++++++++++++++++- > >> hw/vfio/pci.h | 2 ++ > >> 2 files changed, 36 insertions(+), 1 deletion(-) > >> > >> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c > >> index c5e6fe61cb..64cf8e07d9 100644 > >> --- a/hw/vfio/pci.c > >> +++ b/hw/vfio/pci.c > >> @@ -40,6 +40,9 @@ > >> #include "pci.h" > >> #include "trace.h" > >> #include "qapi/error.h" > >> +#include "migration/blocker.h" > >> +#include "qemu/option.h" > >> +#include "qemu/option_int.h" > >> > >> #define TYPE_VFIO_PCI "vfio-pci" > >> #define PCI_VFIO(obj) OBJECT_CHECK(VFIOPCIDevice, obj, TYPE_VFIO_PCI) > >> @@ -2698,6 +2701,12 @@ static void > >> vfio_unregister_req_notifier(VFIOPCIDevice *vdev) > >> vdev->req_enabled = false; > >> } > >> > >> +static int has_net_failover_arg(void *opaque, const char *name, > >> + const char *value, Error **errp) > >> +{ > >> + return (strcmp(name, "net_failover_pair_id") == 0); > >> +} > >> + > >> static void vfio_realize(PCIDevice *pdev, Error **errp) > >> { > >> VFIOPCIDevice *vdev = PCI_VFIO(pdev); > >> @@ -2710,6 +2719,20 @@ static void vfio_realize(PCIDevice *pdev, Error > >> **errp) > >> int groupid; > >> int i, ret; > >> bool is_mdev; > >> + uint16_t class_id; > >> + > >> + if (qemu_opt_foreach(pdev->qdev.opts, has_net_failover_arg, > >> + (void *) pdev->qdev.opts, &err) == 0) { > > > >Why do we need a qemu_opt_foreach here versus testing > >vdev->net_failover_pair_id as you do below or similar to how we test > >sysfsdev immediately below this chunk? > > We don't need it, I will change it and move it to where we check for > the PCI class. > > > >> + error_setg(&vdev->migration_blocker, > >> + "VFIO device doesn't support migration"); > >> + ret = migrate_add_blocker(vdev->migration_blocker, &err); > > > >Where's the migrate_del_blocker()/error_free() for any other realize > >error or device removal? > > > >> + if (err) { > >> + error_propagate(errp, err); > >> + error_free(vdev->migration_blocker); > >> + } > > > >As Connie noted, unclear if this aborts or continues without a > >migration blocker, which would be bad. > > It aborts in my test. PCI realize propagates it further and eventually > it leads to aborting qemu. > > It looks like this now: > > if (!pdev->net_failover_pair_id) { > error_setg(&vdev->migration_blocker, > "VFIO device doesn't support migration"); > ret = migrate_add_blocker(vdev->migration_blocker, &err); > if (err) { > error_propagate(errp, err); > } else { > error_propagate(errp, vdev->migration_blocker); > } > goto error; This unconditionally goes to error when we don't have a failover pair set :-\ I suspect we don't want any sort of error propagate in the success case, the migration_blocker pre-defines the error when the migration is blocked, right? Thanks, Alex > } else { > pdev->qdev.allow_unplug_during_migration = true; > } > > >> + } else { > >> + pdev->qdev.allow_unplug_during_migration = true; > >> + } > >> > >> if (!vdev->vbasedev.sysfsdev) { > >> if (!(~vdev->host.domain || ~vdev->host.bus || > >> @@ -2812,6 +2835,14 @@ static void vfio_realize(PCIDevice *pdev, Error > >> **errp) > >> goto error; > >> } > >> > >> + if (vdev->net_failover_pair_id != NULL) { > >> + class_id = pci_get_word(pdev->config + PCI_CLASS_DEVICE); > >> + if (class_id != PCI_CLASS_NETWORK_ETHERNET) { > >> + error_setg(errp, "failover device is not an Ethernet device"); > >> + goto error; > >> + } > >> + } > > > >Not clear to me why we do this separate from setting up the migration > >blocker or why we use a different mechanism to test for the property. > > I'm moving this check to hw/pci/pci.c as you suggested. > > >> + > >> /* vfio emulates a lot for us, but some bits need extra love */ > >> vdev->emulated_config_bits = g_malloc0(vdev->config_size); > >> > >> @@ -3110,6 +3141,8 @@ static Property vfio_pci_dev_properties[] = { > >> display, ON_OFF_AUTO_OFF), > >> DEFINE_PROP_UINT32("xres", VFIOPCIDevice, display_xres, 0), > >> DEFINE_PROP_UINT32("yres", VFIOPCIDevice, display_yres, 0), > >> + DEFINE_PROP_STRING("net_failover_pair_id", VFIOPCIDevice, > >> + net_failover_pair_id), > > > >Should this and the Ethernet class test be done in PCIDevice? The > >migration aspect is the only thing unique to vfio since we don't > >otherwise support it, right? For instance, I should be able to > >setup an emulated NIC with this failover pair id too, right? Thanks, > > Yes, we can do it in PCIDevice. Using it with an emulated device. > It wouldn't make sense for production but could make sense for > testing purposes. > > Thanks for the review! > > regards, > Jens