On Tue, Apr 13, 2021 at 10:01 AM Marc Zyngier <m...@kernel.org> wrote: > > On Tue, 13 Apr 2021 14:29:32 +0100, > Peter Geis <pgwipe...@gmail.com> wrote: > > > > On Tue, Apr 13, 2021 at 5:23 AM Marc Zyngier <m...@kernel.org> wrote: > > > > > > Hi Peter, > > > > > > On Mon, 12 Apr 2021 21:49:59 +0100, > > > Peter Geis <pgwipe...@gmail.com> wrote: > > > > > > > > Good Afternoon, > > > > > > > > I am assisting with early bringup of the rk3566 based quartz64 > > > > development board for mainline linux. > > > > I've encountered a few issues with allocating ITS on their version of > > > > the GIC-V3. > > > > The first issue is the ITS controller can only use 32bit addresses. > > > > This leads to the following error: > > > > [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 > > > > [ 0.000000] GICv3: GIC: Using split EOI/Deactivate mode > > > > [ 0.000000] GICv3: 320 SPIs implemented > > > > [ 0.000000] GICv3: 0 Extended SPIs implemented > > > > [ 0.000000] GICv3: Distributor has no Range Selector support > > > > [ 0.000000] Root IRQ handler: gic_handle_irq > > > > [ 0.000000] GICv3: 16 PPIs implemented > > > > [ 0.000000] GICv3: CPU0: found redistributor 0 region > > > > 0:0x00000000fd460000 > > > > [ 0.000000] ITS [mem 0xfd440000-0xfd45ffff] > > > > [ 0.000000] ITS@0x00000000fd440000: Devices doesn't stick: > > > > f907000100190600 f907000000190600 > > > > > > Ouch. That looks pretty bad. Bit 32 of the register doesn't stick, and > > > that's right in the middle of the address. The register should be > > > fully writable as far as the address field is concerned. > > > > > > Please dump the distributor and ITS IIDR registers so that I can find > > > the TRM for the exact IP. > > > > Yeah, I'm getting a crash course in how GIC-V3 sets up, so this is a > > learning experience too. > > > > The GICD_IIDR, GICR_IIDR, and GITS_IIDR are all 0x0201743B > > Looks like our mate GIC600 r1p6, which is the most recent revision, > and reasonably bug free AFAIK. Other examples of this GIC integrated > on HW have access to seem to work rather well, and that's with (a lot) > more than 4GB of RAM. > > > > > > > > > > [ 0.000000] ITS@0x00000000fd440000: failed probing (-6) > > > > [ 0.000000] ITS: No ITS available, not enabling LPIs > > > > > > > > Downstream fixed this by adding the GFP_DMA32 flag to the memory > > > > allocations. > > > > > > Urgh... this really looks like broken silicon to me. > > > > I was afraid of that. > > > > > > > > > They also force clear the GITS_BASER_SHAREABILITY_MASK. > > > > > > Why? Does this also apply to the command queue? Are they forcing cache > > > flushing? > > > > The patch doesn't have a description, but the name of the function is > > "force_inner_cache mode". > > It disables shareability and caching, but I've experimented with > > removing this part and it doesn't make a difference in upstream setup. > > (Things are still broken in the same way). It ends up setting > > ITS_FLAGS_CMDQ_NEEDS_FLUSHING as well. > > > > > > > > > Unfortunately while this allowed ITS to allocate on downstream, as > > > > soon as MSIs attempted to use it all interrupts would time out. > > > > > > > > On upstream, we observe this during allocation: > > > > [ 0.000000] ITS [mem 0xfd440000-0xfd45ffff] > > > > [ 0.000000] ITS@0x00000000fd440000: allocated 8192 Devices @3810000 > > > > (indirect, esz 8, psz 64K, shr 1) > > > > [ 0.000000] ITS@0x00000000fd440000: allocated 32768 Interrupt > > > > Collections @3820000 (flat, esz 2, psz 64K, shr 1) > > > > [ 0.000000] GICv3: using LPI property table @0x0000000003830000 > > > > [ 0.000000] GICv3: CPU0: using allocated LPI pending table > > > > @0x0000000003840000 > > > > [ 0.000000] ITS queue timeout (64 1) > > > > [ 0.000000] ITS cmd its_build_mapc_cmd failed > > > > [ 0.000000] ITS queue timeout (96 1) > > > > [ 0.000000] ITS cmd its_build_invall_cmd failed > > > > <snip> > > > > > > So the command queue is not making forward progress. Either because > > > the ITS cannot access the commands, or because it cannot use the > > > memory it has been allocated. Please dump GITS_CBASER (and the value > > > that has been written to it), just in case it shows the same > > > brokenness as the GITS_BASER registers... > > > > It seems yes, CBASER is broken in the same way: > > Value written: b8000001001b040f > > Value read: b8000000001b040f > > Here you go. They haven't only half broken it. Please also check the > GICR_PENDBASER/GICR_PROPBASER values on each RD to see if they have > similar behaviours.
Yes, it seems all accesses are limited to 32bit. > > What happens if you hack all the allocations to happen in the low 4GB > of the PA space? It seems to work correctly. The downstream hacks used GFP_DMA32 which gets discarded by kmalloc_fix_flags on certain allocations. Switching to GFP_DMA seems to have satisfied it, but it feels wrong using this code. Need to check the corner cases to make sure I'm not missing something. > > > > > > > [...] > > > > > > > Any assistance you can provide would be greatly appreciated. > > > > > > I'm not sure there is much we can do without a lot more details about > > > the HW. We need to know the exact GIC implementation they are using > > > (ARM has two versions of the GICv3 IP), and we also need to find out > > > *how* this has been integrated. Only Rockchip can tell you that. > > > > The rk3568 TRM says to reference the > > "ARM_GIC-600_r1p6-00rel0_Technical_Reference_Manual.pdf" for > > information on the GIC, if that helps narrow it down too. > > Here you go: > > https://documentation-service.arm.com/static/5e7ddddacbfe76649ba53034 > > That doesn't tell us what they did to it though, and whether only the > ITS is affected or the RDs are similarly brain-damaged. I guess that's > yet another piece of HW I will not have to spend any money on... > > M. > > -- > Without deviation from the norm, progress is not possible.
