On Sat, Mar 11, 2017 at 10:37:09AM +0000, Mark Cave-Ayland wrote:
> I've been looking at an issue with virtio BARs being mapped incorrectly
> on qemu-system-sparc64, and as part of this investigation found that
> some 64-bit PCI BAR memory regions don't appear correctly in "info mtree".
>
> The easiest way to see this is to launch QEMU as below and then check
> the output of "info mtree":
>
> ./qemu-system-sparc64 -device virtio-net-pci
>
> The interesting part is this:
>
> > address-space: memory
> > 0000000000000000-ffffffffffffffff (prio 0, i/o): system
> > 0000000000000000-0000000007ffffff (prio 0, ram): sun4u.ram
> > 000001fe00000000-000001fe0000ffff (prio 0, i/o): apb-config
> > 000001fe01000000-000001fe01ffffff (prio 0, i/o): apb-pci-config
> > 000001fe02000000-000001fe0200ffff (prio 0, i/o): alias apb-pci ioport
> > @io 0000000000000000-000000000000ffff
> > 000001ff00000000-000001ffffffffff (prio 0, i/o): pci-mmio
> > 000001ff00000000-000001ff000fffff (prio 1, i/o): alias pci_bridge_mem
> > @pci_bridge_pci 0000000000000000-00000000000fffff
> > 000001ff00000000-000001ff000fffff (prio 1, i/o): alias
> > pci_bridge_pref_mem @pci_bridge_pci 0000000000000000-00000000000fffff
> > 000001ff00000000-000001ff000fffff (prio 1, i/o): alias pci_bridge_mem
> > @pci_bridge_pci 0000000000000000-00000000000fffff
> > 000001ff00000000-000001ff000fffff (prio 1, i/o): alias
> > pci_bridge_pref_mem @pci_bridge_pci 0000000000000000-00000000000fffff
> > 000001ff000a0000-000001ff000affff (prio 2, i/o): alias vga.chain4
> > @vga.vram 0000000000000000-000000000000ffff
> > 000001ff000a0000-000001ff000bffff (prio 1, i/o): vga-lowmem
> > 000001ff01000000-000001ff01ffffff (prio 1, ram): vga.vram
> > 000001ff02000000-000001ff02000fff (prio 1, i/o): vga.mmio
> > 000001ff02000400-000001ff0200041f (prio 0, i/o): vga ioports
> > remapped
> > 000001ff02000500-000001ff02000515 (prio 0, i/o): bochs dispi
> > interface
> > 000001ff02000600-000001ff02000607 (prio 0, i/o): qemu extended regs
> > 000001ff02010000-000001ff0201ffff (prio 1, rom): vga.rom
> > 000001ff03000000-000001ff03ffffff (prio 1, i/o): alias bar0 @io
> > 0000000000000000-0000000000ffffff
> > 000001ff04000000-000001ff0403ffff (prio 1, rom): ne2000.rom
> > 000001ff04080000-000001ff040bffff (prio 1, rom): virtio-net-pci.rom
> > 000001fe04040000-000001fe04043fff (prio 1, i/o): virtio-pci
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> Here virtio-pci is mapped to address 0x1fe04040000 even though its
> parent pci-mmio memory region is at 0x1ff00000000.
>
> > 000001fe04040000-000001fe04040fff (prio 0, i/o): virtio-pci-common
> > 000001fe04041000-000001fe04041fff (prio 0, i/o): virtio-pci-isr
> > 000001fe04042000-000001fe04042fff (prio 0, i/o): virtio-pci-device
> > 000001fe04043000-000001fe04043fff (prio 0, i/o): virtio-pci-notify
> > 000001fff0000000-000001fff03fffff (prio 0, rom): sun4u.prom
>
> The bug in the current OpenBIOS is that 64-bit BARs are incorrectly
> identified, and so the BAR MSB register is set to 0xffffffff to detect
> the size as per a standard 32-bit BAR. At this point OpenBIOS detects
> something is wrong and aborts, leaving the 64-bit BAR set to 0xffffffff
> 0x04040000.
>
> This is accurately reflected in the virtio-pci memory region itself, i.e.
>
> > memory-region: virtio-pci
> > ffffffff04040000-ffffffff04043fff (prio 1, i/o): virtio-pci
> > ffffffff04040000-ffffffff04040fff (prio 0, i/o): virtio-pci-common
> > ffffffff04041000-ffffffff04041fff (prio 0, i/o): virtio-pci-isr
> > ffffffff04042000-ffffffff04042fff (prio 0, i/o): virtio-pci-device
> > ffffffff04043000-ffffffff04043fff (prio 0, i/o): virtio-pci-notify
>
> But as you can see above the overall effect is that the virtio-pci
> device ends up somehow being mapped outside (and indeed, before the
> start of) its parent memory region.
And specifically we have:
000001ff00000000 + ffffffff04040000 = 000001fe04043000
which seems to be where this is coming from.
It does have to indicate some kind of modelling problem:
since it looks like on sparc BAR is an offset within
PCI MMIO region, I would expect an offset value bigger than
the region size not to be visible.
Specifically documentation states:
- all direct subregions of the root region are matched against the address, in
descending priority order
- if the address lies outside the region offset/size, the subregion is
discarded
> Fixing the bug in OpenBIOS itself is fairly trivial, although it took
> much longer to diagnose due to the misleading output above. It does seem
> that something isn't quite right with the handling of the large 64-bit
> BARs in this case, so Michael suggested I email the list for further
> comment.
>
>
> ATB,
>
> Mark.
