On 2012-01-13 22:56, Alex Williamson wrote: > On Fri, 2012-01-13 at 22:33 +0100, Jan Kiszka wrote: >> On 2012-01-13 22:05, Alex Williamson wrote: >>> On Fri, 2012-01-13 at 22:00 +0100, Jan Kiszka wrote: >>>> On 2012-01-04 04:21, Alex Williamson wrote: >>>>> On Mon, 2011-12-19 at 19:49 +0530, Shashidhar Patil wrote: >>>>>> Hi, >>>>>> I am running Ubuntu 10.10 (amd64) on a 2 socket nehalem based >>>>>> server with IOH 5520. 5520 supports VTD. >>>>>> I enabled DMAR with intel_iommu=on. The box has intel 82599 adapter >>>>>> which I assigned through VT-D to FreeBSD 8.2 running >>>>>> as guest os. The ixgbe driver detects the device and the driver >>>>>> successfully configures the device. But the link >>>>>> never comes up. It looks like link up/down interrupts are not >>>>>> delivered. Then I checked kvm interrupt assignment and as expected >>>>>> kvm could not make MSI-X entries for the VT-d guest. So no output from >>>>>> "grep kvm /proc/interrupt". By enabling some debugs in the >>>>>> qemu-kvm I figured out that the MSI-x updates are not received >>>>>> properly. It does look like Linux updates MSI-X table in a batch >>>>>> fashion >>>>>> which qemu-kvm gets in one shot and every thing works fine in case of >>>>>> linux. In case of FreeBSD PCIE updates come /MSI-X entry >>>>>> which qemu-kvm can't make use. >>>>> >>>>> That's right, Linux and Windows both seem to setup the MSI-X table then >>>>> enable it in one shot, so we only trigger the interrupt programming when >>>>> the enable bit is set. We don't trigger changes on writes to the MSI-X >>>>> table... not very accurate emulation of mask bits. >>>> >>>> According to the PCI spec, updates that happen while a vector is >>>> unmasked, need not be considered by the hardware (thus the hypervisor >>>> here). Is that the scenario here? >>> >>> I'm assuming the vector is masked in the MSI-X table. So Linux/Windows >>> do: >>> >>> a) program MSI-X table >>> b) enable MSI-X in capability register >>> >>> Whereas FreeBSD does: >>> >>> a) enable MSI-X in capability register (vectors masked in table) >>> b) program and unmask individual vectors >> >> That should work with the current code. It checks the number of vectors >> on each config write, iterates the whole table, and then updates the > ^^^^^^^^^^^^^^^^^^^^ >> kernel configuration accordingly. It even requires the enable bit in the >> cap register to be set before doing this. > > That's the problem, we only do it on config writes overlapping the MSI-X > flags. We don't do anything for writes to the MSI-X table. It might be > as simple as calling assigned_dev_update_msix() from msix_mmio_writel() > when the mask bit is toggled. I'm not sure what might fall out of that > though.
Ah indeed. Now I recall to have fixed this in my MSI-X refactoring series. I introduced config notifiers that are triggered by the MSI-X layer on every relevant modification, and the device assignment code hook the update function into this. I really need to dig into that series soon again and refresh it. In the meantime, we could try what you suggest (if the cap enable bit is set). Jan
signature.asc
Description: OpenPGP digital signature
