On Tue, 2 Jan 2018 18:35:49 +0100
Torbjorn Jansson <torbjorn.jans...@mbox200.swipnet.se> wrote:

> On 2018-01-02 12:41, Doug Johnson wrote:
> > 
> > Hi All,
> > 
> > I have an HD 6990 card which is a dual gpu card from 2011 (pre GCN) that I 
> > am trying to pass through with vfio.
> > 
> > I successfully passed both GPUs to a VM with QEMU, but seabios gets stuck 
> > on an option ROM (which I believe is attached to the second GPU). I can see 
> > the card initialize, though, so I know that works. If I specify an empty 
> > ROM file to the second GPU, the BIOS boots and I can get into windows. I 
> > can install drivers, but it appears that they only use one GPU (so, I 
> > cannot enable crossfile between them). I also downloaded a copy of the ROM 
> > file for my model (Gigabyte) from techpowerup.com and get the same behavior 
> > of getting stuck if I specify the file on both GPUs. The AMD drivers also, 
> > notably, don't seem to recognize the card properly saying the generic name 
> > of "HD 6900 Series," which I believe isn't the correct behavior, although I 
> > haven't tried it in a real windows machine to verify yet.
> > 
> > I suspect the issue may have to do with a PLX bridge chip that shows up in 
> > lspci (which is part of the card) but that isn't directly passed to the VM, 
> > and the GPU bios hangs trying to do something with that, but I am not 
> > entirely sure.
> > 
> > Attached is the output of my lspci -knn as well as my VM spinup script and 
> > the output of seabios debug... Any other relevant system details can be 
> > provided.
> > 
> > I am not sure where to go from here... Perhaps emulating the PCI topology 
> > of the card somehow? Maybe even there is a way to pass the PLX bridge chip?
> > 
> > Thanks for your help in advance
> 
> first of all i have no experience with radeon cards but as far as i know it 
> is 
> not possible to get crossfire/sli working in a virtual machine.
> 
> also running cards with built in uefi support and os that uses uefi have a 
> higher chance of working since you avoid a lot of old legacy stuff.

SO I came across this blog post: 
https://nils.schimmelmann.us/post/154884462507/gaming-with-xdma-crossfire-passthrough-on-ubuntu

Which deals with getting crossfire working with XDMA which is a GCN 1.1 
feature. The setup in the post also requires UEFI but alas this card is too old 
so I have to use legacy BIOS. I did noticed however the requirement that each 
GPU have its own root port switch:

-device 
ioh3420,port=0xe0,chassis=1,id=pcie.1,bus=pcie.0,multifunction=on,addr=0x1c \
-device 
ioh3420,port=0x18,chassis=2,id=pcie.2,bus=pcie.0,multifunction=on,addr=0x3 \

So I went ahead and modified my script with the following:

 -device 
ioh3420,port=0xe0,chassis=1,id=pcie.1,bus=pcie.0,multifunction=on,addr=0x1c \
 -device 
ioh3420,port=0xe0,chassis=2,id=pcie.2,bus=pcie.0,multifunction=on,addr=0x3 \
 -device vfio-pci,host=08:00.0,bus=pcie.1,addr=00.0,multifunction=on,x-vga=on \
 -device vfio-pci,host=08:00.1,bus=pcie.1,addr=00.1 \
 -device vfio-pci,host=09:00.0,bus=pcie.2,addr=00.0,multifunction=on,romfile= \
 -device vfio-pci,host=09:00.1,bus=pcie.2,addr=00.1 \

And to my surprise after windows booted and reinstalled the drivers with the 
new topology, it actually works. The drivers recognize one GPU as 'primary' and 
the other as 'linked.' I still had to specify an empty romfile on the secondary 
GPU, though, to avoid the lockup in seabios. I ran some benchmarks and it looks 
like the two GPUs are in fact running in tandem. 

-- 
Doug Johnson <dou...@dougvj.net>

Attachment: pgpiwezuQINGJ.pgp
Description: PGP signature

_______________________________________________
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Reply via email to