When bridging traffic across two Intel X540-AT2 NICs using openvswitch(OVS), I can get ~10 Mpps of small packet (64 byte) traffic on a Dell R620 system using 2x Intel E5-2650 v2 @ 2.6 GHz using a Spirent generator.
Using the identical system and after enabling intel_iommu=on during boot, the throughput drops to ~500 Kpps. Looking at the output from perf, I see that ~80% of the CPUs are consumed spinning on locks. The highest contention appears to be on async_umap_flush_lock acquired in add_unmap. This area of code appears unchanged since it was introduced back in 2008. The code review back then called out the lack of scalability of the code as seen in this snippet from http://lkml.iu.edu/hypermail/linux/kernel/0802.3/2944.html : ---begin quote--- ... >> +static void add_unmap(struct dmar_domain *dom, struct iova *iova) >> +{ >> + unsigned long flags; >> + >> + spin_lock_irqsave(&async_umap_flush_lock, flags); > > How scalable is this? Its not very, but I'm doing an insert walk of a list and need the locks. ... ---end quote--- >From looking at bugzilla no one else appears to have reported this issue so I'm wondering if there is an easy way to side step this - is there a workaround so that I don't need to use "intel_iommu=on"? I'm testing with the "intel_iommu=on" setting so that I can get SR-IOV NICs passed through to VMs running over KVM. Is this possible any other way? Let me know if you want me to open a bug on bugzilla and I can upload the perf.data and perf report output there instead of attaching large files to this email. For now, if it helps, here's the snippet from the top of perf report: ---begin extract from perf report--- ... 5.43% ksoftirqd/0 [kernel.kallsyms] [k] _raw_spin_lock_irqsave | --- _raw_spin_lock_irqsave | |--62.66%-- add_unmap | | | |--99.93%-- intel_unmap_page.part.40 | | intel_unmap_page | | ixgbe_poll | | net_rx_action | | __do_softirq ... ---end extract from perf report--- I’m not on the iommu alias - so I’d appreciate being cc’ed on any replies. Thanks, Charles _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
