On Fri, 2014-08-01 at 18:39 +0200, Jan Kiszka wrote: > On 2014-08-01 18:35, Alex Williamson wrote: > > On Fri, 2014-08-01 at 09:25 -0600, Alex Williamson wrote: > >> On Fri, 2014-08-01 at 09:35 +0800, Le Tan wrote: > >>> Hi Alex, > >>> > >>> 2014-07-30 22:46 GMT+08:00 Alex Williamson <alex.william...@redhat.com>: > >>>> On Wed, 2014-07-30 at 22:16 +0800, Le Tan wrote: > >>>>> Hi Michael, > >>>>> > >>>>> 2014-07-30 21:16 GMT+08:00 Michael S. Tsirkin <m...@redhat.com>: > >>>>>> On Wed, Jul 30, 2014 at 08:24:04PM +0800, Le Tan wrote: > >>>>>>> Hi, > >>>>>>> I am testing vfio in L1 with my VT-d emulation project. I assigned one > >>>>>>> of the two AHCI controllers in L1 to L2 via vfio. After I ran the QEMU > >>>>>>> in L1, it complains that: > >>>>>>> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no > >>>>>>> available reset mechanism. > >>>>>>> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no > >>>>>>> available reset mechanism. > >>>>>>> > >>>>>>> Then L2 paused when the SeaBIOS executed in ahci_controller_setup(). I > >>>>>>> look into this and found that: > >>>>>>> val = ahci_ctrl_readl(ctrl, HOST_CTL); > >>>>>>> ahci_ctrl_writel(ctrl, HOST_CTL, val | HOST_CTL_AHCI_EN); > >>>>>>> When the BIOS tried to read the HOST_CTL, it returns 0x80000002, which > >>>>>>> bit 2 (Interrupt Enable) is 1. The AHCI manual says that this bit > >>>>>>> should be cleared by default. So maybe L1 didn't reset the device > >>>>>>> before assigning it to L2? > >>>>>>> Then the BIOS tried to write back to HOST_CTL and it was stuck here. > >>>>>>> :( > >>>>>>> > >>>>>>> So can anyone give me some advice? About the state of PCI device or > >>>>>>> bus-level reset? > >>>>>>> > >>>>>>> Here is the detail of the environment and the way I did the vfio. > >>>>>>> 1. lspci in L1 said: > >>>>>>> 00:03.0 SATA controller [0106]: Intel Corporation 82801IR/IO/IH > >>>>>>> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02) > >>>>>>> 00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC > >>>>>>> Interface Controller [8086:2918] (rev 02) > >>>>>>> 00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH > >>>>>>> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02) > >>>>>>> 00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus > >>>>>>> Controller [8086:2930] (rev 02) > >>>>>>> 2. Unbind 00:03.0 and do vfio: > >>>>>>> modprobe -r vfio_iommu_type1 > >>>>>>> modprobe vfio_iommu_type1 allow_unsafe_interrupts=1 > >>>>>>> modprobe vfio-pci > >>>>>>> echo 0000:00:03.0 > /sys/bus/pci/devices/0000\:00\:03.0/driver/unbind > >>>>>>> echo "8086 2922" > /sys/bus/pci/drivers/vfio-pci/new_id > >>>>>>> 3. run L2 with "-device vfio-pci,host=00:03.0" > >>>>>>> > >>>>>>> Any help is appreciated! Thanks very much! > >>>>>>> > >>>>>>> Regards, > >>>>>>> Le > >>>>>> > >>>>>> Clearly, bus level reset can't work for the root bus :) > >>>>> > >>>>> Thanks very much! > >>>>> I test the vfio with a second-bus ahci controller and it didn't > >>>>> complain about the lack of reset mechanism. :) And the return val of > >>>>> HOST_CTL is normal now (the same as emulated ahci controller). > >>>>> However, it still paused when the BIOS tried to write to the HOST_CTL. > >>>>> Do you have any idea? > >>>>> And we should just test vfio and legacy pci-assignment with second bus > >>>>> devices, not considering the root-bus devices? > >>>> > >>>> AHCI seems like a poor choice of device for this work, they typically > >>>> don't support any kind of reset and they can be troublesome even for the > >>>> L1 assignment. You really want something with FLR support so that both > >>>> the host and L1 guest can potentially reset the device. That said, you > >>>> may still run into bugs with a L1 guest directed FLR. Thanks, > >>>> > >>>> Alex > >>>> > >>> > >>> So what device do you think is suitable for the pci-assign test? e1000? > >>> I just tested it with sound card ac97 and USB controller. But I don't > >>> know how to attach them to a pcie-to-pci bridge, so maybe they weren't > >>> reset before being assigned to L2. But it seems that they can work. > >>> 1. With the sound card, I assigned it to L2 via both vfio and legacy > >>> pci-assign and I could hear the music played in L2 from host's > >>> speakers. Of course, the vfio also complained about the lack of reset > >>> mechanism. > >> > >> First off, maybe I'm a little confused, are you trying to assigned > >> emulated devices for the L1 guest to the L2 guest or are these assigned > >> devices to the L1 guest that you want to re-assign to L2? AFAIK, we > >> don't have any emulated devices that support reset, but it wouldn't be > >> that hard to add a PM capability to one that advertises a soft reset on > >> D3hot->D0 transition and calls the QEMU driver reset function when that > >> occurs. This would provide the most flexibility. > > > > Or even better might be to add an Advanced Features capability to each > > device so it can do an explicit FLR. We don't tend to trust PM reset as > > being reliably on all devices. > > But FLR is PCIe-only, isn't it?
Nope, according to the ECN it's intended for conventional PCI devices on the PCIe root complex where a complete conversion to PCIe cannot be justified. Maybe it's not 100% to the letter of the capability to add it to a random non-RC device, but it seems like a good solution here. > Also, it may let some of our device > models deviate from their real versions (I suppose, e.g., none of the > e1000 devices we currently emulate exposed FLR). Of course, but what are the chances that the driver will care? > In any case, I suggested Le to focus on his GSoC task again (VT-d > emulation) and postpone this topic to later point (likely outside GSoC). > Only few weeks left... Ok. Thanks, Alex
