Hello!
I need some help to split my IOMMU groups, let’s start from the beginning. I have an Asrock J3455-ITX motherboard with the latest Proxmox version. On the motherboard there are an onboard NIC, an external NIC (plugged into the only one PCIe slot) and an external Wifi card (plugged into the M.2 slot). On the host proxmox I have a pfSense running in a VM and I would like to pass through my external NIC and the Wifi card but they are in the same IOMMU group than the onbord NIC. See my dmesg and lspci output: # dmesg |grep -i iommu [ 0.000000] Command line: BOOT_IMAGE=/ROOT/pve-1@/boot/vmlinuz-4.15.18-10-pve root=ZFS=rpool/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on iommu=pt pcie_acs_override=downstream [ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA [ 0.000000] Kernel command line: BOOT_IMAGE=/ROOT/pve-1@/boot/vmlinuz-4.15.18-10-pve root=ZFS=rpool/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on iommu=pt pcie_acs_override=downstream [ 0.000000] DMAR: IOMMU enabled [ 0.004000] DMAR-IR: IOAPIC id 1 under DRHD base 0xfed65000 IOMMU 1 [ 1.340272] iommu: Adding device 0000:00:00.0 to group 0 [ 1.340287] iommu: Adding device 0000:00:02.0 to group 1 [ 1.340306] iommu: Adding device 0000:00:0e.0 to group 2 [ 1.340325] iommu: Adding device 0000:00:0f.0 to group 3 [ 1.340339] iommu: Adding device 0000:00:12.0 to group 4 [ 1.340384] iommu: Adding device 0000:00:13.0 to group 5 [ 1.340408] iommu: Adding device 0000:00:13.1 to group 5 [ 1.340427] iommu: Adding device 0000:00:13.2 to group 5 [ 1.340447] iommu: Adding device 0000:00:13.3 to group 5 [ 1.340465] iommu: Adding device 0000:00:15.0 to group 6 [ 1.340487] iommu: Adding device 0000:00:1f.0 to group 7 [ 1.340500] iommu: Adding device 0000:00:1f.1 to group 7 [ 1.340515] iommu: Adding device 0000:01:00.0 to group 5 [ 1.340527] iommu: Adding device 0000:02:00.0 to group 5 [ 1.340538] iommu: Adding device 0000:03:00.0 to group 5 [ 1.340549] iommu: Adding device 0000:04:00.0 to group 5 # lspci -tv -[0000:00]-+-00.0 Intel Corporation Atom/Celeron/Pentium Processor N4200/N3350/E3900 Series Host Bridge +-02.0 Intel Corporation Device 5a85 +-0e.0 Intel Corporation Atom/Celeron/Pentium Processor N4200/N3350/E3900 Series Audio Cluster +-0f.0 Intel Corporation Atom/Celeron/Pentium Processor N4200/N3350/E3900 Series Trusted Execution Engine +-12.0 Intel Corporation Atom/Celeron/Pentium Processor N4200/N3350/E3900 Series SATA AHCI Controller +-13.0-[01]----00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller +-13.1-[02]----00.0 Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter +-13.2-[03]----00.0 ASMedia Technology Inc. ASM1062 Serial ATA Controller +-13.3-[04]----00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller +-15.0 Intel Corporation Atom/Celeron/Pentium Processor N4200/N3350/E3900 Series USB xHCI +-1f.0 Intel Corporation Atom/Celeron/Pentium Processor N4200/N3350/E3900 Series Low Pin Count Interface \-1f.1 Intel Corporation Atom/Celeron/Pentium Processor N4200/N3350/E3900 Series SMBus Controller # lspci -vvnnns 00:13.3 00:13.3 PCI bridge [0604]: Intel Corporation Atom/Celeron/Pentium Processor N4200/N3350/E3900 Series PCI Express Port A #4 [8086:5adb] (rev fb) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin D routed to IRQ 125 Bus: primary=00, secondary=04, subordinate=04, sec-latency=0 I/O behind bridge: 0000c000-0000cfff Memory behind bridge: 91300000-913fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #6, Speed 5GT/s, Width x1, ASPM not supported, Exit Latency L0s <1us, L1 <4us ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #5, PowerLimit 10.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState+ RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee002d8 Data: 0000 Capabilities: [90] Subsystem: ASRock Incorporation Atom/Celeron/Pentium Processor N4200/N3350/E3900 Series PCI Express Port A [1849:5adb] Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [100 v0] #00 Capabilities: [140 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd- EgressCtrl- DirectTrans- ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- Capabilities: [150 v0] #00 Capabilities: [200 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ PortCommonModeRestoreTime=40us PortTPowerOnTime=10us L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=0ns L1SubCtl2: T_PwrOn=10us Kernel driver in use: pcieport Kernel modules: shpchp What I already did and tried: – Updated the BIOS and turned on intel virtualization (VT-x, VT-d) – updated the grub with intel_iommu=on iommu=pt pcie_acs_override=downstream (tried downstream,multifunction as well) – loaded vfio, vfio_iommu_type1, vfio_pci, vfio_virqfd kernel modules at boot time (/etc/modules) – write: options vfio_iommu_type1 allow_unsafe_interrupts=1 to /etc/modprobe.d/iommu_unsafe_interrupts.conf – write: options vfio-pci ids=[8086:5ad9],[8086:5ada] to /etc/modprobe.d/vfio.conf Do you have any idea what could be the problem? Thank you
_______________________________________________ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users