On Tue, 2 Jan 2018 18:35:49 +0100 Torbjorn Jansson <torbjorn.jans...@mbox200.swipnet.se> wrote:
> On 2018-01-02 12:41, Doug Johnson wrote: > > > > Hi All, > > > > I have an HD 6990 card which is a dual gpu card from 2011 (pre GCN) that I > > am trying to pass through with vfio. > > > > I successfully passed both GPUs to a VM with QEMU, but seabios gets stuck > > on an option ROM (which I believe is attached to the second GPU). I can see > > the card initialize, though, so I know that works. If I specify an empty > > ROM file to the second GPU, the BIOS boots and I can get into windows. I > > can install drivers, but it appears that they only use one GPU (so, I > > cannot enable crossfile between them). I also downloaded a copy of the ROM > > file for my model (Gigabyte) from techpowerup.com and get the same behavior > > of getting stuck if I specify the file on both GPUs. The AMD drivers also, > > notably, don't seem to recognize the card properly saying the generic name > > of "HD 6900 Series," which I believe isn't the correct behavior, although I > > haven't tried it in a real windows machine to verify yet. > > > > I suspect the issue may have to do with a PLX bridge chip that shows up in > > lspci (which is part of the card) but that isn't directly passed to the VM, > > and the GPU bios hangs trying to do something with that, but I am not > > entirely sure. > > > > Attached is the output of my lspci -knn as well as my VM spinup script and > > the output of seabios debug... Any other relevant system details can be > > provided. > > > > I am not sure where to go from here... Perhaps emulating the PCI topology > > of the card somehow? Maybe even there is a way to pass the PLX bridge chip? > > > > Thanks for your help in advance > > first of all i have no experience with radeon cards but as far as i know it > is > not possible to get crossfire/sli working in a virtual machine. > > also running cards with built in uefi support and os that uses uefi have a > higher chance of working since you avoid a lot of old legacy stuff. SO I came across this blog post: https://nils.schimmelmann.us/post/154884462507/gaming-with-xdma-crossfire-passthrough-on-ubuntu Which deals with getting crossfire working with XDMA which is a GCN 1.1 feature. The setup in the post also requires UEFI but alas this card is too old so I have to use legacy BIOS. I did noticed however the requirement that each GPU have its own root port switch: -device ioh3420,port=0xe0,chassis=1,id=pcie.1,bus=pcie.0,multifunction=on,addr=0x1c \ -device ioh3420,port=0x18,chassis=2,id=pcie.2,bus=pcie.0,multifunction=on,addr=0x3 \ So I went ahead and modified my script with the following: -device ioh3420,port=0xe0,chassis=1,id=pcie.1,bus=pcie.0,multifunction=on,addr=0x1c \ -device ioh3420,port=0xe0,chassis=2,id=pcie.2,bus=pcie.0,multifunction=on,addr=0x3 \ -device vfio-pci,host=08:00.0,bus=pcie.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=08:00.1,bus=pcie.1,addr=00.1 \ -device vfio-pci,host=09:00.0,bus=pcie.2,addr=00.0,multifunction=on,romfile= \ -device vfio-pci,host=09:00.1,bus=pcie.2,addr=00.1 \ And to my surprise after windows booted and reinstalled the drivers with the new topology, it actually works. The drivers recognize one GPU as 'primary' and the other as 'linked.' I still had to specify an empty romfile on the secondary GPU, though, to avoid the lockup in seabios. I ran some benchmarks and it looks like the two GPUs are in fact running in tandem. -- Doug Johnson <dou...@dougvj.net>
pgpiwezuQINGJ.pgp
Description: PGP signature
_______________________________________________ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
