Hi,
> -----Ursprüngliche Nachricht-----
> Von: Julien Grall <jul...@xen.org>
> Gesendet: Montag, 23. November 2020 19:27
> An: Leo Krueger <leo.krue...@zal.aero>; Stefano Stabellini
> <stefano.stabell...@xilinx.com>
> Cc: Peng Fan <peng....@nxp.com>; bru...@xilinx.com; Cornelia Bruelhart
> <cornelia.bruelh...@zal.aero>; oleksandr_andrushche...@epam.com; xen-
> de...@lists.xenproject.org; bertrand.marq...@arm.com
> Betreff: Re: AW: AW: AW: AW: AW: Xen data from meta-virtualization layer
>
>
>
> On 22/11/2020 22:55, Leo Krueger wrote:
> > Hi Julien,
>
> Hi Leo,
>
> >
> > finally I could try out what you suggested, please find my answers inline.
>
> Thank you for sending the logs!
>
> >
> >> -----Ursprüngliche Nachricht-----
> >> Von: Julien Grall <jul...@xen.org>
> >> Gesendet: Mittwoch, 18. November 2020 13:24
> >> An: Stefano Stabellini <stefano.stabell...@xilinx.com>; Leo Krueger
> >> <leo.krue...@zal.aero>
> >> Cc: Peng Fan <peng....@nxp.com>; bru...@xilinx.com; Cornelia
> >> Bruelhart <cornelia.bruelh...@zal.aero>;
> >> oleksandr_andrushche...@epam.com; xen- de...@lists.xenproject.org;
> >> bertrand.marq...@arm.com
> >> Betreff: Re: AW: AW: AW: AW: Xen data from meta-virtualization layer
> >>
> >> Hi,
> >>
> >> On 17/11/2020 23:53, Stefano Stabellini wrote:
> >>> Adding Bertrand, Oleksandr, Julien, and others -- they have a more
> >>> recent experience with GICv3 ITS than me and might be able to help.
> >>> I am attaching the device tree Leo sent a few days ago for reference.
> >>>
> >>>
> >>> Typically when you can set the ethernet link up and no packets are
> >>> exchanged it is because of a missing interrupt. In this case a
> >>> missing MSI.
> >>>
> >>> Bertrand, I believe you tried the GIC ITS driver with PCI devices
> >>> recently. It is expected to work correctly with MSIs in Dom0, right?
> >>
> >> OSSTest has some hardware (e.g. Thunder-X) where ITS is required to
> >> boot Dom0. I haven't seen any failure on recent Xen. We are testing
> >> 4.11 and onwards on Thunder-X.
> >>
> >> However, it may be possible that some more work is necessary for
> >> other hardware (e.g. workaround, missing code...). See more below.
> >>
> >>>
> >>>
> >>>
> >>> On Tue, 17 Nov 2020, Leo Krueger wrote:
> >>>> Hi,
> >>>>
> >>>> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
> >>>> before...) but then had to add the following node to my device tree
> >>>>
> >>>> gic_lpi_base: syscon@0x80000000 {
> >>>> compatible = "gic-lpi-base";
> >>
> >> I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo,
> >> could you clarify which flavor/version of Linux you are using?
> >
> > It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.
>
> Do you have a link to the Linux tree? Is there any additional patches on top
> of
> vanilla?
Linux tree is found here:
https://github.com/kontron/linux-smarc-sal28/commits/master-LSDK-19.09
(up to the latest commit in that branch)
>
> > While searching around the Internet for any solution, I came across [0]
> which contained the gic-lpi-base node.
> > So I just tried adding it (quite desperate I know) and voila, it at least
> brought me one step further (XEN exposing the ITS)...
>
> I am slightly confused to how this would help. Xen and, AFAICT, Linux don't
> understand gic-lpi-base. Do you have modification in your Linux to use it?
I have no idea, to be honest. Maybe it is about the memory being reserved due
to that node or something like that?
>
> Looking at the DT changes in [0], it looks like the node is not a child of
> gic@.
> So I think Xen will map the region to Dom0.
>
> There are two things that I can notice:
> 1) This region is RAM, but I can't find any reserve node. Is there any
> specific
> code in Linux to reserve it?
> 2) The implementation in U-boot seems to suggest that the firmware will
> configure the LPIs and then enable it. If that's the case, then Xen needs to
> re-use the table in the DT rather than allocating a new one.
> However, I would have expected an error message in the log:
>
> "GICv3: CPUx: Cannot initialize LPIs"
>
> At least Xen should not expose gic-lpi-base to the kernel, but I will wait on
> more details about the Linux kernel used before commenting more.
>
> I would also be interested to know more details about the failure when gic-
> lpi-base is not added in your DT. In particular, I am interested to understand
> why Xen would not expose the ITS as we don't parse that node.
How can I supply you with more information in regard to that? Without that
node, ITS was not exposed at all.
>
> [...]
>
> > For XEN 4.13.2 I had to adapt your patch slightly [1], see below (yes I
> > know,
> quite ugly in parts).
>
> No worries, debug patches are not meant to be nice to read ;).
>
> > Find attached the boot log and an output of "xl dmesg" which is truncated
> due to the large amount of messages.
> >
> > When enabling the network interface (gbe0), the following output is
> visible:
> >
> > root@kontron-sal28:~# ip link set up dev gbe0
> > (XEN) vgic-v3-its.c:902:d0v0 vITS cmd 0x0c: 000000170000000c
> > 0000000000000001 0000000000000000 0000000000000000
> > (XEN) vgic-v3-its.c:902:d0v0 vITS cmd 0x05: 0000000000000005
> > 0000000000000000 0000000000000000 0000000000000000
>
> 0xc is INV and 0x5 is SYNC. Most likely the driver unmask the interrupt by
> writing in the property table (access are not trapped to Xen) and then
> requested to invalidate the cache state.
>
> > [ 34.034598] Atheros 8031 ethernet 0000:00:00.3:05: attached PHY driver
> [Atheros 8031 ethernet] (mii_bus:phy_addr=0000:00:00.3:05, irq=POLL)
> > [ 34.041111] 8021q: adding VLAN 0 to HW filter on device gbe0
> > [ 34.041209] IPv6: ADDRCONF(NETDEV_UP): gbe0: link is not ready
> > root@kontron-sal28:~# [ 35.041951] fsl_enetc 0000:00:00.0 gbe0: Link is
> Down
> > [ 38.114426] fsl_enetc 0000:00:00.0 gbe0: Link is Up - 1Gbps/Full - flow
> control off
> > [ 38.114508] IPv6: ADDRCONF(NETDEV_CHANGE): gbe0: link becomes
> ready
> >
> > Does that tell you anything?
>
> It is at least a good sign because it means Linux is able to initialize/talk
> to the
> vITS.
>
> I would lean towards one (or multiple) issue with pITS and/or the device-tree
> exposed to Linux. I am not entirely what exactly... I think having more
> details
> about the Linux setup would be helpful.
Ok let me know what you need from my side. I was considering the following
things to try out next:
- a more recent u-boot version as this might fix problems with the msi-map (at
least that is what I think, I am not an expert here)
- a different device tree (a more recent one, ...)
- ...
>
> I will reply on Rahul's e-mail separately.
>
> Cheers,
Best wishes!
>
> --
> Julien Grall
