Hi Folks,
Long time listener (~2days), first time...
Based on years of pain-free experience with VF NICs, I naively thought
that installing a second GPU in a desktop for VM pass through would be a
straight forward task. This has turned out to be an incorrect
assumption. I have two desktop systems with [unfortunately] the same
basic configuration:
* supermicro x10sae
* "Intel(R) Xeon(R) CPU E3-1276 v3 @ 3.60GHz"
* Fedora 25 / linux 4.10
In one of these systems I have installed 2x nvidia gtx 1070 GPUs:
$ uname -r
4.10.5-200.fc25.x86_64
$ lspci -nn | grep -i vga
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104
[GeForce GTX 1070] [10de:1b81] (rev a1)
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104
[GeForce GTX 1070] [10de:1b81] (rev a1)
$ find /sys/kernel/iommu_groups/ -type l | egrep '0[12]:00'
/sys/kernel/iommu_groups/1/devices/0000:02:00.1
/sys/kernel/iommu_groups/1/devices/0000:01:00.1
/sys/kernel/iommu_groups/1/devices/0000:02:00.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
and both GPUs are landing in the same vfio group. :( Digging into the
motherboard manual shows that there is some sort of PCIe switch between
the physical PCIe slots and the CPU's root port:
pg 16 - http://www.supermicro.com/manuals/motherboard/C226/MNL-1479.pdf
I have been completely unable to find a datasheet for the ASM1480
"switch". The homepage isn't helpful:
http://www.asmedia.com.tw/eng/e_show_products.php?item=109
I briefly explored the ACS override patch but the second quirks.c hunk
does not apply to vanilla linux 4.9/4.10 sources. I am rather hesitant
to try to forward port as I know nothing about the pci subsystem and the
file has a changed a lot since the patch epoc. I took it as a bad sign
that google can't find a forward port of the patch for current kernels.
However, Google did eventually turn up a BZ post from Alex Williamson
stating that Xeon E3s, at least as of "v3", do not support ACS:
https://bugzilla.redhat.com/show_bug.cgi?id=1113399#c9
https://www-ssl.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e3-1200v3-spec-update.pdf
So it sounds like I'm looking at new hardware to get this working with
stock Fedora kernels?
I prefer E3s for a desktop CPU as they have ECC memory support. It took
forever to dig through Intel's website to find basic CPU datasheets.
The E3-1200 v5 datasheets don't seem to mention ACS and the E3-1200 v6
datasheets (current product generation) haven't been published yet (!).
http://www.intel.com/content/www/us/en/processors/xeon/xeon-technical-resources.html
Does anyone know definitely if the E3-1200 v6 generation supports ACS in
the on-die PCIe controller?
My guess is this is not the case or it would be called out as a "new
security feature" in the marketing material. However, I am still hoping
to stay with an E3-1200 CPU in order to keep costs low. It appears that
the supermicro x11sat-f motherboard has a "PLX8747" switch between the
CPU root port and the physical slots:
pg 18 - http://www.supermicro.com/manuals/motherboard/H4/MNL-1823.pdf
Which I think may be a broadcom "pex 8747":
https://www.broadcom.com/products/pcie-switches-bridges/pcie-switches/pex8747
which does claim ACS support in the overview sheet. Does anyone know if
ACS is indeed working with the "pex 8747" and/or the sm x1?sat series of
motherboards? Alternatively, does anyone know of an E3-1200 motherboard
that does have working ACS with 2x [8x/16x] PCIe slots?
If anyone has followed my rambling this far, thank you for your extreme
patience. I would like to thank Alex Williamson for his vfio blog and
the kvmforum talk up on youtube -- I would be completely lost at this
point without that information.
Cheers,
-Josh
--
_______________________________________________
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users
