On Mon, 8 Jun 2020, CodeWiz2280 wrote:
> It actually shows only 1 interrupt for any of the devices in that list
> (e.g. spi, ttyS0, ethernet) so you're probably right on the money with
> it being an interrupt acknowledge issue. Any help you can provide is
> greatly appreciated.
>
> On Mon, Jun 8, 2020 at 4:40 AM Bertrand Marquis
> <bertrand.marq...@arm.com> wrote:
> >
> >
> >
> > > On 5 Jun 2020, at 20:12, CodeWiz2280 <codewiz2...@gmail.com> wrote:
> > >
> > > On Fri, Jun 5, 2020 at 11:05 AM CodeWiz2280 <codewiz2...@gmail.com> wrote:
> > >>
> > >> On Fri, Jun 5, 2020 at 8:47 AM Bertrand Marquis
> > >> <bertrand.marq...@arm.com> wrote:
> > >>>
> > >>>
> > >>>
> > >>>> On 5 Jun 2020, at 13:42, CodeWiz2280 <codewiz2...@gmail.com> wrote:
> > >>>>
> > >>>> On Fri, Jun 5, 2020 at 8:30 AM Julien Grall <jul...@xen.org> wrote:
> > >>>>>
> > >>>>> Hi,
> > >>>>>
> > >>>>> On 05/06/2020 13:25, CodeWiz2280 wrote:
> > >>>>>> The Keystone uses the netcp driver, which has interrupts from 40-79
> > >>>>>> listed in the device tree (arch/arm/boot/keystone-k2e-netcp.dtsi).
> > >>>>>> I'm using the same device tree between my non-xen standalone kernel
> > >>>>>> and my dom0 kernel booted by xen. In the standalone (non-xen) kernel
> > >>>>>> the ethernet works fine, but I don't see any of its interrupts in the
> > >>>>>> output of /proc/iomem. I'm not seeing them in /proc/iomem when
> > >>>>>> running dom0 under Xen either. When booting with Xen I get this
> > >>>>>> behavior where the ifconfig output shows 1 RX message and 1 TX
> > >>>>>> message, and then nothing else.
> > >>>>>
> > >>>>> I am not sure whether this is a typo in the e-mail. /proc/iomem is
> > >>>>> listing the list of the MMIO regions. You want to use
> > >>>>> /proc/interrupts.
> > >>>>>
> > >>>>> Can you confirm which path you are dumping?
> > >>>> Yes, that was a typo. Sorry about that. I meant that I am dumping
> > >>>> /proc/interrupts and do not
> > >>>> see them under the non-xen kernel or xen booted dom0.
> > >>>
> > >>> Could you post both /proc/interrupts content ?
> > >>
> > >> Standalone non-xen kernel (Ethernet works)
> > >> # cat /proc/interrupts
> > >> CPU0 CPU1 CPU2 CPU3
> > >> 17: 0 0 0 0 GICv2 29 Level
> > >> arch_timer
> > >> 18: 9856 1202 457 650 GICv2 30 Level
> > >> arch_timer
> > >> 21: 0 0 0 0 GICv2 142 Edge
> > >> timer-keystone
> > >> 22: 0 0 0 0 GICv2 52 Edge
> > >> arm-pmu
> > >> 23: 0 0 0 0 GICv2 53 Edge
> > >> arm-pmu
> > >> 24: 0 0 0 0 GICv2 54 Edge
> > >> arm-pmu
> > >> 25: 0 0 0 0 GICv2 55 Edge
> > >> arm-pmu
> > >> 26: 0 0 0 0 GICv2 36 Edge
> > >> 26202a0.keystone_irq
> > >> 27: 1435 0 0 0 GICv2 309 Edge
> > >> ttyS0
> > >> 29: 0 0 0 0 GICv2 315 Edge
> > >> 2530000.i2c
> > >> 30: 1 0 0 0 GICv2 318 Edge
> > >> 2530400.i2c
> > >> 31: 0 0 0 0 GICv2 321 Edge
> > >> 2530800.i2c
> > >> 32: 69 0 0 0 GICv2 324 Edge
> > >> 21000400.spi
> > >> 33: 0 0 0 0 GICv2 328 Edge
> > >> 21000600.spi
> > >> 34: 0 0 0 0 GICv2 332 Edge
> > >> 21000800.spi
> > >> 70: 0 0 0 0 GICv2 417 Edge
> > >> ks-pcie-error-irq
> > >> 79: 0 0 0 0 PCI-MSI 0 Edge
> > >> PCIe PME, aerdrv
> > >> 88: 57 0 0 0 GICv2 80 Level
> > >> hwqueue-528
> > >> 89: 57 0 0 0 GICv2 81 Level
> > >> hwqueue-529
> > >> 90: 47 0 0 0 GICv2 82 Level
> > >> hwqueue-530
> > >> 91: 41 0 0 0 GICv2 83 Level
> > >> hwqueue-531
> > >> IPI0: 0 0 0 0 CPU wakeup interrupts
> > >> IPI1: 0 0 0 0 Timer broadcast
> > >> interrupts
> > >> IPI2: 730 988 1058 937 Rescheduling
> > >> interrupts
> > >> IPI3: 2 3 4 6 Function call
> > >> interrupts
> > >> IPI4: 0 0 0 0 CPU stop interrupts
> > >> IPI5: 0 0 0 0 IRQ work interrupts
> > >> IPI6: 0 0 0 0 completion interrupts
> > >>
> > >> Xen dom0 (Ethernet stops)
> > >> # cat /proc/interrupts
> > >> CPU0
> > >> 18: 10380 GIC-0 27 Level arch_timer
> > >> 19: 0 GIC-0 142 Edge timer-keystone
> > >> 20: 88 GIC-0 16 Level events
> > >> 21: 0 xen-dyn Edge -event xenbus
> > >> 22: 0 GIC-0 36 Edge 26202a0.keystone_irq
> > >> 23: 1 GIC-0 312 Edge ttyS0
> > >> 25: 1 GIC-0 318 Edge
> > >> 27: 1 GIC-0 324 Edge 21000400.spi
> > >> 28: 0 GIC-0 328 Edge 21000600.spi
> > >> 29: 0 GIC-0 332 Edge 21000800.spi
> > >> 65: 0 GIC-0 417 Edge ks-pcie-error-irq
> > >> 74: 0 PCI-MSI 0 Edge PCIe PME, aerdrv
> > >> 83: 1 GIC-0 80 Level hwqueue-528
> > >> 84: 1 GIC-0 81 Level hwqueue-529
> > >> 85: 1 GIC-0 82 Level hwqueue-530
> > >> 86: 1 GIC-0 83 Level hwqueue-531
> > >> 115: 87 xen-dyn Edge -virq hvc_console
> > >> IPI0: 0 CPU wakeup interrupts
> > >> IPI1: 0 Timer broadcast interrupts
> > >> IPI2: 0 Rescheduling interrupts
> > >> IPI3: 0 Function call interrupts
> > >> IPI4: 0 CPU stop interrupts
> > >> IPI5: 0 IRQ work interrupts
> > >> IPI6: 0 completion interrupts
> > >> Err: 0
> > > After getting a chance to look at this a little more, I believe the
> > > TX/RX interrupts for the ethernets map like this:
> > >
> > > eth0 Rx - hwqueue-528
> > > eth1 Rx - hwqueue-529
> > > eth0 Tx - hwqueue-530
> > > eth1 Tx - hwqueue-531
> > >>
> > > The interrupt counts in the standlone working kernel seem to roughly
> > > correspond to the counts of Tx/Rx messages in ifconfig. Going on
> > > that, its clear that only 1 interrupt has been received for Tx and 1
> > > for Rx in the Xen Dom0 equivalent. Any thoughts on this?
> >
> > This definitely look like an interrupt acknowledgement issue.
> > This could be caused by 2 things I remember of:
> > - front vs level interrupts
> > - a problem with forwarded interrupt acknowledgement.
> > I think there was something related to that where the vcpu ack was not
> > properly
> > handled on a keystone and I had to change the way the interrupt was acked
> > for
> > forwarded hardware interrupts.
Is there maybe some sort of secondary interrupt controller (secondary in
addition to the GIC) or interrupt "concentrator" on KeyStone?
Or is it just a small deviation from normal GIC behavior?
