On 18/07/2017 0:50, Kinsella, Ray wrote:
Hi folks,
Hi Ray,
I am trying to create a VM that supports hot-plugging a large number of virtio-net-pci device, up to 1000 devices initially. From the docs (see below) and from playing with QEMU, it looks like there are two options. Both with limitations. PCI Express switch It looks like using a PCI Express switch hierarchy is not an option due to bus exhaustion. Each downstream port creates a separate bus and each downstream port only supports hot-plugging a single device. So this gives us a max of 256-ish buses/devices pairs.
Right.
PCI Root Ports The other option is use a flatter hierarchy, with a number of multi-function PCI Root Ports hanging off 'pcie.0'. However each 'PCI Root Port' can support hot-plugging a single device. So this method really becomes a function of how many free address we have on 'pci.0'. If we make room for say 16 multifunction devices, we get 16*8 ... 128 So ultimately, this will approach will give us a similar to number to using a switch. Is there another method?
You can use multi-function PCIe Root Ports, this will give you 8 ports per slot, if you have 16 empty slots (I think we have more) you reach 128 root ports. Then you can use multi-function virtio-net-pci devices, this will give you 8 functions per port, so you reach the target of 1024 devices. You loose hot-plug granularity since you can hot-plug 8-functions group, but maybe is OK, depending on your scenario.
( pxb-pcie doesn't support hotplug for instance, and only a single pcie domain is supported qemu )
Even so, you can use one cold-plugged pxb-pcie if you don't have enough empty slots on pcie.0, in order to reach the maximum number of PCIe Root Ports (256) which is the maximum for a single PCI domain. If you need granularity per single device (1000+ hot-pluggable), you could enhance the pxb-pcie to support multiple pci domains. Thanks, Marcel
Thanks, Ray K Pcie.txt, ( https://github.com/qemu/qemu/blob/master/docs/pcie.txt ) Q35 preso ( http://wiki.qemu.org/images/4/4e/Q35.pdf )
