On 20/07/2017 3:44, Kinsella, Ray wrote:
Hi Marcel,
Hi Ray,
You can use multi-function PCIe Root Ports, this will give you 8 ports
per slot, if you have 16 empty slots (I think we have more) you reach
128 root ports.
Then you can use multi-function virtio-net-pci devices, this will
give you 8 functions per port, so you reach the target of 1024 devices.
You loose hot-plug granularity since you can hot-plug 8-functions group,
but maybe is OK, depending on your scenario.
Thanks for the advice losing the hotplug granularity is something I
think I can live with. It would mean, I would have to track how many
ports are allocated to a VM, and create 8 new ports when 1 is required,
caching the other 7 for when they are needed.
Even so, you can use one cold-plugged pxb-pcie if you don't
have enough empty slots on pcie.0, in order to reach the maximum
number of PCIe Root Ports (256) which is the maximum for a single
PCI domain.
Took your advice see the attached cfg, it works exactly as you
indicated. If you are interested, you can use it from your VM adding
-readconfig to your qemu cmd line. I can currently only manage to start
a VM with around 50 coldplugged virtio devices before something breaks.
That's strange. Please ensure the virtio devices are working in
virtio 1.0 mode (disable-modern=0,disable-legacy=1).
Let us know any problems you see.
Not sure what yet, I will try scaling it with hotplugging tomorrow.
Updates?
If you need granularity per single device (1000+ hot-pluggable),
you could enhance the pxb-pcie to support multiple pci domains.
Do think there would be much work in this?
Not really. All you have to do is to add a property to the pxb-pci/pxb
devices: pci_domain=x; then update the ACPI table to include the pxb
domain. You also have to tweak a little the pxb-pcie/pxb devices
to not share the bus numbers if pci_domain > 0.
Thanks,
Marcel
Thanks,
Ray K
