On 09/01/2015 04:56 PM, Konrad Rzeszutek Wilk wrote:
On Tue, Sep 01, 2015 at 05:39:46PM +0000, Justin Acker wrote:
Taking this to the dev list from users.
Is there a way to force or enable pirq delivery to a set of cpus as opposed to
single device from being a assigned a single pirq so that its interrupt can be
distributed across multiple cpus? I believe the device drivers do support
multiple queues when run natively without the Dom0 loaded. The device in
question is the xhci_hcd driver for which I/O transfers seem to be slowed when
the Dom0 is loaded. The behavior seems to pass through to the DomU if pass
through is enabled. I found some similar threads, but most relate to Ethernet
controllers. I tried some of the x2apic and x2apic_phys dom0 kernel arguments,
but none distributed the pirqs. Based on the reading relating to IRQs for Xen,
I think pinning the pirqs to cpu0 is done to avoid an interrupt storm. I tried
IRQ balance and when configured/adjusted it will balance individual pirqs, but
not multiple interrupts.
Yes. You can do it with smp affinity:
https://cs.uwaterloo.ca/~brecht/servers/apic/SMP-affinity.txt
I don't follow the "behavior seems to pass through to the DomU if pass through is
enabled" ?
With irqbalance enabled in Dom0:
What version? There was a bug in it where it would never distribute the IRQs
properly
across the CPUs.
Boris (CC-ed) might remember the upstream patch that made this work properly?
I think we ended up taking the latest version of irqbalance as the one
that added support for Xen guests was still not quite working. Besides,
that patch was for xen-dyn-events, not for pirqs.
-boris
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
CPU6 CPU7
76: 11304 0 149579 0 0 0
0 0 xen-pirq-msi 0000:00:1f.2
77: 1243 0 0 35447 0 0
0 0 xen-pirq-msi radeon
78: 82521 0 0 0 0 0
0 0 xen-pirq-msi xhci_hcd
79: 23 0 0 0 0 0
0 0 xen-pirq-msi mei_me
80: 11 0 0 0 0 741
0 0 xen-pirq-msi em1
81: 350 0 0 0 1671 0
0 0 xen-pirq-msi iwlwifi
82: 275 0 0 0 0 0
0 0 xen-pirq-msi snd_hda_intel
With native 3.19 kernel:
Without Dom0 for the same system from the first message:
# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
CPU6 CPU7
0: 33 0 0 0 0 0
0 0 IR-IO-APIC-edge timer
8: 0 0 0 0 0 0
1 0 IR-IO-APIC-edge rtc0
9: 20 0 0 0 0 1
1 1 IR-IO-APIC-fasteoi acpi
16: 15 0 8 1 4 1
1 1 IR-IO-APIC 16-fasteoi ehci_hcd:usb3
18: 703940 5678 1426226 1303 3938243 111477
757871 510 IR-IO-APIC 18-fasteoi ath9k
23: 11 2 3 0 0 17
2 0 IR-IO-APIC 23-fasteoi ehci_hcd:usb4
24: 0 0 0 0 0 0
0 0 DMAR_MSI-edge dmar0
25: 0 0 0 0 0 0
0 0 DMAR_MSI-edge dmar1
26: 20419 1609 26822 567 62281 5426
14928 395 IR-PCI-MSI-edge 0000:00:1f.2
27: 17977230 628258 44247270 120391 1597809883 14440991
152189328 73322 IR-PCI-MSI-edge xhci_hcd
28: 563 0 0 0 1 0
6 0 IR-PCI-MSI-edge i915
29: 14 0 0 4 2 4
0 0 IR-PCI-MSI-edge mei_me
30: 39514 1744 60339 157 129956 19702
72140 83 IR-PCI-MSI-edge eth0
31: 3 0 0 1 54 0
0 2 IR-PCI-MSI-edge snd_hda_intel
32: 28145 284 53316 63 139165 4410
25760 27 IR-PCI-MSI-edge eth1-rx-0
33: 1032 43 2392 5 1797 265
1507 20 IR-PCI-MSI-edge eth1-tx-0
34: 0 1 0 0 0 1
2 0 IR-PCI-MSI-edge eth1
35: 5 0 0 12 148 6
2 1 IR-PCI-MSI-edge snd_hda_intel
The USB controller is an Intel C210:
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family
USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI])
Subsystem: Dell Device 053e
Flags: bus master, medium devsel, latency 0, IRQ 78
Memory at f7f20000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [70] Power Management version 2
Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci
On Tuesday, September 1, 2015 11:50 AM, Ian Campbell
<ian.campb...@citrix.com> wrote:
On Tue, 2015-09-01 at 13:56 +0000, Justin Acker wrote:
Thanks Ian,
I appreciate the explanation. I believe the device drivers do support
multiple queues when run natively without the Dom0 loaded. The device in
question is the xhci_hcd driver for which I/O transfers seem to be slowed
when the Dom0 is loaded. The behavior seems to pass through to the DomU
if pass through is enabled. I found some similar threads, but most relate
to Ethernet controllers. I tried some of the x2apic and x2apic_phys dom0
kernel arguments, but none distributed the pirqs. Based on the reading
relating to IRQs for Xen, I think pinning the pirqs to cpu0 is done to
avoid an I/O storm. I tried IRQ balance and when configured/adjusted it
will balance individual pirqs, but not multiple interrupts.
Is there a way to force or enable pirq delivery to a set of cpus as you
mentioned above or omit a single device from being a assigned a PIRQ so
that its interrupt can be distributed across all cpus?
A PIRQ is the way an interrupt is exposed to a PV guest, without it there
would be no interrupt at all.
I'm afraid I'm out of my depth WRT how x86/MSIs and Xen x86/PV pirqs
interact, in particular WRT configuring which set of CPUs can have the IRQ
delivered.
If no one else chimes in soon I'd suggest taking this to the dev list, at
the very least someone who knows what they are talking about (i.e. other
than me) might be able to help.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
