On Tue, 19 Dec 2017 00:55:32 +1100 Alexey Kardashevskiy <a...@ozlabs.ru> wrote:
> On 19/12/17 00:28, Alex Williamson wrote: > > On Mon, 18 Dec 2017 20:04:23 +1100 > > Alexey Kardashevskiy <a...@ozlabs.ru> wrote: > > > >> On 18/12/17 16:02, Alex Williamson wrote: > >>> With recently proposed kernel side vfio-pci changes, the MSI-X vector > >>> table area can be mmap'd from userspace, allowing direct access to > >>> non-MSI-X registers within the host page size of this area. However, > >>> we only get that direct access if QEMU isn't also emulating MSI-X > >>> within that same page. For x86/64 host, the system page size is 4K > >>> and the PCI spec recommends a minimum of 4K to 8K alignment to > >>> separate MSI-X from non-MSI-X registers, therefore only devices which > >>> don't honor this recommendation would see any improvement from this > >>> option. The real targets for this feature are hosts where the page > >>> size exceeds the PCI spec recommended alignment, such as ARM64 systems > >>> with 64K pages. > >>> > >>> This new x-msix-relocation option accepts the following options: > >>> > >>> off: Disable MSI-X relocation, use native device config (default) > >>> auto: Automaically relocate MSI-X MMIO to another BAR or offset > >>> based on minimum additional MMIO requirement > >>> bar0..bar5: Specify the target BAR, which will either be extended > >>> if the BAR exists or added if the BAR slot is available. > >> > >> > >> While I am digesting the patchset, here are some test results. > > > > Thanks for testing! > > > >> This is the device: > >> > >> 00:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 > >> PCI-Express Fusion-MPT SAS-3 (rev 02) > > > > BAR1: > > > >> Memory at 210000000000 (64-bit, non-prefetchable) [size=64K] > > > > BAR3: > > > >> Memory at 210000040000 (64-bit, non-prefetchable) [size=256K] > >> > >> Capabilities: [c0] MSI-X: Enable+ Count=96 Masked- > >> Vector table: BAR=1 offset=0000e000 > >> PBA: BAR=1 offset=0000f000 > >> > >> > >> Test #1: x-msix-relocation = "off": > >> > >> FlatView #1 > >> AS "memory", root: system > >> AS "cpu-memory", root: system > >> Root memory region: system > >> 0000000000000000-000000007fffffff (prio 0, ram): ppc_spapr.ram > >> 0000210000000000-000021000000dfff (prio 0, i/o): 0001:03:00.0 BAR 1 > >> 000021000000e000-000021000000e5ff (prio 0, i/o): msix-table > >> 000021000000e600-000021000000ffff (prio 0, i/o): 0001:03:00.0 BAR 1 > >> @000000000000e600 > >> 0000210000040000-000021000007ffff (prio 0, ramd): 0001:03:00.0 BAR 3 > >> mmaps[0] > >> > >> Ok, works. > >> > >> > >> Test #2: x-msix-relocation = "auto": > >> > >> FlatView #2 > >> AS "memory", root: system > >> AS "cpu-memory", root: system > >> Root memory region: system > >> 0000000000000000-000000007fffffff (prio 0, ram): ppc_spapr.ram > >> 0000200080000000-00002000800005ff (prio 0, i/o): msix-table > >> 0000200080000600-000020008000ffff (prio 1, i/o): 0001:03:00.0 base BAR 0 > >> @0000000000000600 > >> 0000210000000000-000021000000ffff (prio 0, i/o): 0001:03:00.0 BAR 1 > >> 0000210000040000-000021000007ffff (prio 0, ramd): 0001:03:00.0 BAR 3 > >> mmaps[0] > >> > >> > >> The guest fails probing because the first 64bit BAR is broken. > >> > >> lspci: > >> > >> Region 0: Memory at 200080000000 (32-bit, prefetchable) [size=64K] > >> Region 1: Memory at 210000000000 (64-bit, non-prefetchable) [size=64K] > >> Region 3: Memory at 210000040000 (64-bit, non-prefetchable) [size=256K] > >> > >> Capabilities: [c0] MSI-X: Enable- Count=96 Masked- > >> Vector table: BAR=0 offset=00000000 > >> PBA: BAR=0 offset=00000600 > > > > Why do you suppose it's broken? The added BAR0 is 32bit, it cannot be > > 64bit since BAR1 is implemented. I don't see anything fundamentally > > different between this and the working BAR5 test below. > > > BAR1 (0x14..0x17) uses BAR0 (0x10..0x13) as upper 32bits when it is 64bit > BAR, no? AIUI, if BAR1 is 64bit, it consumes 0x14-0x17 for the lower 32bis and 0x18-1b for the upper 32bits, ie. it consumes BAR1 + BAR2. Likewise the 64bit BAR3 also consumes BAR4. See for instance the 82576 datasheet: https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/82576eb-gigabit-ethernet-controller-datasheet.pdf 9.4.11.2 shows the BAR configuration in 64bit mode, 64bit BAR0 consumes BAR0 (lower) + BAR1 (upper), 64bit BAR2 consumes BAR2 (lower) + BAR3 (upper), and the MSI-X BAR becomes 64bit at BAR4, consuming BAR4 (lower) + BAR5 (upper). lspci would show this as Region 0, 2, 4. The layout of your SAS card does seem poorly thought out that they've essentially precluded a 3rd 64bit BAR by starting with BAR1, but perhaps it's for compatibility with an equally poorly designed 32bit version of the device. Thanks, Alex
