There's rarely much harm to hardware in playing around with anything other than the fan control / bios / soldering, though obviously you can kill your OS installation (but that's just one reason why we love VM's!). I wouldn't expect much success as the four network devices are likely together because they share resource/control functions. As you say - you will have to trick the system into thinking they are isolated using the ACS isolation patch. Maybe somebody more experienced can give a clearer answer though as I've never used that patch but I'm deeply inclined to recommend a virtualised solution as it's robust .. scalable and far easier to maintain.

For example, just run the 4 network cards on the host and make virtual networks or bridge for the clients. Routing is a complex problem and has to be handled somehwere so it's better you know where plus with 4 network cards I can imagine the the performance impact (latency or throughput) of using virtual nets/bridges on all computers/routers will round off to 0 when measured against the overhead introduced by the cards not having unique 8GT/s connections to their destinations.

I can think of cases where it would be worth the effort but I suggest you have a dig around in some of the network virtualisation documents (arch is quite a good place to start if I remember). It gets very scary very quickly even before you try to pass them through separately and if you've got 20,000 of these you'll need more knowledgeable help than me.

On 26/03/16 11:28, YAEGASHI Takeshi wrote:
Hello,

I have a fitlet http://www.fit-pc.com/web/products/fitlet/ with 4 GbE
(I211) ports and wanted to passthrough some of them to KVM guests, ie
assign one I211 to each guests.  I've tried various configurations
with libvirt/kvm but no luck so far.

After reading http://vfio.blogspot.jp/2014/08/iommu-groups-inside-and-out.html
I've got sure that there's a problem in IOMMU groups.  Actually all of
I211 and AMD's PCI bridge [1022:156b] share the same iommu_group 2.

$ lspci -tv
-[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD] Device 1566
            +-00.2  Advanced Micro Devices, Inc. [AMD] Device 1567
            +-01.0  Advanced Micro Devices, Inc. [AMD/ATI] Mullins [Radeon R6 
Graphics]
            +-01.1  Advanced Micro Devices, Inc. [AMD/ATI] Kabini HDMI/DP Audio
            +-02.0  Advanced Micro Devices, Inc. [AMD] Device 156b
            +-02.2-[01]----00.0  Intel Corporation I211 Gigabit Network 
Connection
            +-02.3-[02]----00.0  Intel Corporation I211 Gigabit Network 
Connection
            +-02.4-[03]----00.0  Intel Corporation I211 Gigabit Network 
Connection
            +-02.5-[04]----00.0  Intel Corporation I211 Gigabit Network 
Connection
            +-08.0  Advanced Micro Devices, Inc. [AMD] Device 1537
            +-10.0  Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller
            +-11.0  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller 
[AHCI mode]
            +-12.0  Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller
            +-13.0  Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller
            +-14.0  Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
            +-14.2  Advanced Micro Devices, Inc. [AMD] FCH Azalia Controller
            +-14.3  Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
            +-14.7  Advanced Micro Devices, Inc. [AMD] FCH SD Flash Controller
            +-18.0  Advanced Micro Devices, Inc. [AMD] Device 1580
            +-18.1  Advanced Micro Devices, Inc. [AMD] Device 1581
            +-18.2  Advanced Micro Devices, Inc. [AMD] Device 1582
            +-18.3  Advanced Micro Devices, Inc. [AMD] Device 1583
            +-18.4  Advanced Micro Devices, Inc. [AMD] Device 1584
            \-18.5  Advanced Micro Devices, Inc. [AMD] Device 1585

$ find /sys/kernel/iommu_groups -type l
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/2/devices/0000:00:02.2
/sys/kernel/iommu_groups/2/devices/0000:00:02.3
/sys/kernel/iommu_groups/2/devices/0000:00:02.4
/sys/kernel/iommu_groups/2/devices/0000:00:02.5
/sys/kernel/iommu_groups/2/devices/0000:01:00.0
/sys/kernel/iommu_groups/2/devices/0000:02:00.0
/sys/kernel/iommu_groups/2/devices/0000:03:00.0
/sys/kernel/iommu_groups/2/devices/0000:04:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:08.0
/sys/kernel/iommu_groups/4/devices/0000:00:10.0
/sys/kernel/iommu_groups/5/devices/0000:00:11.0
/sys/kernel/iommu_groups/6/devices/0000:00:12.0
/sys/kernel/iommu_groups/7/devices/0000:00:13.0
/sys/kernel/iommu_groups/8/devices/0000:00:14.0
/sys/kernel/iommu_groups/8/devices/0000:00:14.2
/sys/kernel/iommu_groups/8/devices/0000:00:14.3
/sys/kernel/iommu_groups/8/devices/0000:00:14.7
/sys/kernel/iommu_groups/9/devices/0000:00:18.0
/sys/kernel/iommu_groups/9/devices/0000:00:18.1
/sys/kernel/iommu_groups/9/devices/0000:00:18.2
/sys/kernel/iommu_groups/9/devices/0000:00:18.3
/sys/kernel/iommu_groups/9/devices/0000:00:18.4
/sys/kernel/iommu_groups/9/devices/0000:00:18.5

I'm running Ubuntu 14.04 with xenial kernel 4.4.0-13, libvirt 1.2.2,
qemu 2.0.0.  Using vfio-pci simply failed:

qemu-system-x86_64: -device 
vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: error, group 2 is 
not viable, please ensure all devices within the iommu_group are bound to their 
vfio bus driver.
qemu-system-x86_64: -device 
vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to get group 
2
qemu-system-x86_64: -device 
vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x3: Device initialization 
failed.
qemu-system-x86_64: -device 
vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x3: Device 'vfio-pci' could 
not be initialized

Using legacy KVM device assignment with <driver name='kvm'/> also
failed with unclear reason "Invalid argument":

qemu-system-x86_64: -device 
pci-assign,configfd=24,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x2: Failed to assign 
device "hostdev0" : Invalid argument
qemu-system-x86_64: -device 
pci-assign,configfd=24,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x2: Device 
initialization failed.
qemu-system-x86_64: -device 
pci-assign,configfd=24,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x2: Device 
'kvm-pci-assign' could not be initialized

After some googling I've learned that legacy KVM assignment also
refuses to work for this configuration since kernel 4.2
https://lkml.org/lkml/2015/11/12/661 .  With older kernel 3.19 this
configuration worked as expected.

Is there any chance for me to achieve single I211 passthrough with
recent kernels?  Because the hardware has no flexibility on the bus
topologies, the only possible way would be patching the kernel (ACS
override or PCI quirks).  Is it safe?  Does anyone have any info on
the IOMMU capability of device/bridge/chipset in question?

Regards,

_______________________________________________
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Reply via email to