> On 5 Jun 2020, at 20:12, CodeWiz2280 <codewiz2...@gmail.com> wrote:
>
> On Fri, Jun 5, 2020 at 11:05 AM CodeWiz2280 <codewiz2...@gmail.com> wrote:
>>
>> On Fri, Jun 5, 2020 at 8:47 AM Bertrand Marquis
>> <bertrand.marq...@arm.com> wrote:
>>>
>>>
>>>
>>>> On 5 Jun 2020, at 13:42, CodeWiz2280 <codewiz2...@gmail.com> wrote:
>>>>
>>>> On Fri, Jun 5, 2020 at 8:30 AM Julien Grall <jul...@xen.org> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> On 05/06/2020 13:25, CodeWiz2280 wrote:
>>>>>> The Keystone uses the netcp driver, which has interrupts from 40-79
>>>>>> listed in the device tree (arch/arm/boot/keystone-k2e-netcp.dtsi).
>>>>>> I'm using the same device tree between my non-xen standalone kernel
>>>>>> and my dom0 kernel booted by xen. In the standalone (non-xen) kernel
>>>>>> the ethernet works fine, but I don't see any of its interrupts in the
>>>>>> output of /proc/iomem. I'm not seeing them in /proc/iomem when
>>>>>> running dom0 under Xen either. When booting with Xen I get this
>>>>>> behavior where the ifconfig output shows 1 RX message and 1 TX
>>>>>> message, and then nothing else.
>>>>>
>>>>> I am not sure whether this is a typo in the e-mail. /proc/iomem is
>>>>> listing the list of the MMIO regions. You want to use /proc/interrupts.
>>>>>
>>>>> Can you confirm which path you are dumping?
>>>> Yes, that was a typo. Sorry about that. I meant that I am dumping
>>>> /proc/interrupts and do not
>>>> see them under the non-xen kernel or xen booted dom0.
>>>
>>> Could you post both /proc/interrupts content ?
>>
>> Standalone non-xen kernel (Ethernet works)
>> # cat /proc/interrupts
>> CPU0 CPU1 CPU2 CPU3
>> 17: 0 0 0 0 GICv2 29 Level
>> arch_timer
>> 18: 9856 1202 457 650 GICv2 30 Level
>> arch_timer
>> 21: 0 0 0 0 GICv2 142 Edge
>> timer-keystone
>> 22: 0 0 0 0 GICv2 52 Edge
>> arm-pmu
>> 23: 0 0 0 0 GICv2 53 Edge
>> arm-pmu
>> 24: 0 0 0 0 GICv2 54 Edge
>> arm-pmu
>> 25: 0 0 0 0 GICv2 55 Edge
>> arm-pmu
>> 26: 0 0 0 0 GICv2 36 Edge
>> 26202a0.keystone_irq
>> 27: 1435 0 0 0 GICv2 309 Edge ttyS0
>> 29: 0 0 0 0 GICv2 315 Edge
>> 2530000.i2c
>> 30: 1 0 0 0 GICv2 318 Edge
>> 2530400.i2c
>> 31: 0 0 0 0 GICv2 321 Edge
>> 2530800.i2c
>> 32: 69 0 0 0 GICv2 324 Edge
>> 21000400.spi
>> 33: 0 0 0 0 GICv2 328 Edge
>> 21000600.spi
>> 34: 0 0 0 0 GICv2 332 Edge
>> 21000800.spi
>> 70: 0 0 0 0 GICv2 417 Edge
>> ks-pcie-error-irq
>> 79: 0 0 0 0 PCI-MSI 0 Edge
>> PCIe PME, aerdrv
>> 88: 57 0 0 0 GICv2 80 Level
>> hwqueue-528
>> 89: 57 0 0 0 GICv2 81 Level
>> hwqueue-529
>> 90: 47 0 0 0 GICv2 82 Level
>> hwqueue-530
>> 91: 41 0 0 0 GICv2 83 Level
>> hwqueue-531
>> IPI0: 0 0 0 0 CPU wakeup interrupts
>> IPI1: 0 0 0 0 Timer broadcast interrupts
>> IPI2: 730 988 1058 937 Rescheduling interrupts
>> IPI3: 2 3 4 6 Function call interrupts
>> IPI4: 0 0 0 0 CPU stop interrupts
>> IPI5: 0 0 0 0 IRQ work interrupts
>> IPI6: 0 0 0 0 completion interrupts
>>
>> Xen dom0 (Ethernet stops)
>> # cat /proc/interrupts
>> CPU0
>> 18: 10380 GIC-0 27 Level arch_timer
>> 19: 0 GIC-0 142 Edge timer-keystone
>> 20: 88 GIC-0 16 Level events
>> 21: 0 xen-dyn Edge -event xenbus
>> 22: 0 GIC-0 36 Edge 26202a0.keystone_irq
>> 23: 1 GIC-0 312 Edge ttyS0
>> 25: 1 GIC-0 318 Edge
>> 27: 1 GIC-0 324 Edge 21000400.spi
>> 28: 0 GIC-0 328 Edge 21000600.spi
>> 29: 0 GIC-0 332 Edge 21000800.spi
>> 65: 0 GIC-0 417 Edge ks-pcie-error-irq
>> 74: 0 PCI-MSI 0 Edge PCIe PME, aerdrv
>> 83: 1 GIC-0 80 Level hwqueue-528
>> 84: 1 GIC-0 81 Level hwqueue-529
>> 85: 1 GIC-0 82 Level hwqueue-530
>> 86: 1 GIC-0 83 Level hwqueue-531
>> 115: 87 xen-dyn Edge -virq hvc_console
>> IPI0: 0 CPU wakeup interrupts
>> IPI1: 0 Timer broadcast interrupts
>> IPI2: 0 Rescheduling interrupts
>> IPI3: 0 Function call interrupts
>> IPI4: 0 CPU stop interrupts
>> IPI5: 0 IRQ work interrupts
>> IPI6: 0 completion interrupts
>> Err: 0
> After getting a chance to look at this a little more, I believe the
> TX/RX interrupts for the ethernets map like this:
>
> eth0 Rx - hwqueue-528
> eth1 Rx - hwqueue-529
> eth0 Tx - hwqueue-530
> eth1 Tx - hwqueue-531
>>
> The interrupt counts in the standlone working kernel seem to roughly
> correspond to the counts of Tx/Rx messages in ifconfig. Going on
> that, its clear that only 1 interrupt has been received for Tx and 1
> for Rx in the Xen Dom0 equivalent. Any thoughts on this?
This definitely look like an interrupt acknowledgement issue.
This could be caused by 2 things I remember of:
- front vs level interrupts
- a problem with forwarded interrupt acknowledgement.
I think there was something related to that where the vcpu ack was not properly
handled on a keystone and I had to change the way the interrupt was acked for
forwarded hardware interrupts.
I will try to get more info on that one as I have no access to the code anymore.
Regards
Bertrand
