On 05/09/2016 10:54 AM, Murali Karicheri wrote:
> Francois,
>
> Thanks for responding.
>
> On 05/07/2016 04:35 AM, Francois Romieu wrote:
>> Murali Karicheri <m-kariche...@ti.com> :
>> [...]
>>> I am trying to integrate the rtl8168 PCIe card to have Ethernet functional
>>> on my Keystone EVM.
>>
>> Which (EVM) one ?
> K2G EVM. This is a new EVM for which work has been started to add it to
> upstream
> kernel. But it not there yet. The PCIe h/w in the K2G SoC is a re-use from
> existing SoC such as K2E. The PCIe controller (Root complex a.k.a RC) driver
> based on this PCIe h/w works fine on K2E EVM with a Marvel SATA controller
> connected to the PCIe port. On K2G EVM, I have a PCIe slot to which I have
> plugged the rtl8168 PCIe card and power on with the log I had sent you. The
> serdes driver is responsible for link setup seems to work fine, link is up and
> could detect the rtl8168 below.
>>
>>> I purchased the rtl8111c Gib card from Amazon. The Card is detected
>>> by the RC and I can see it is enumerated and show up when doing lspci
>>> command.
>>
>> What does "the RC" mean ? A different PC ?
>
> RC - Root Complex (PCIe controller - drivers/pci/host/pci-keystone.c, a
> designware
> based PCIe controller driver)
>
> The the PCIe controller driver bindings...
>
> pcie0_phy: phy@2320000 {
> #phy-cells = <0>;
> compatible = "ti,keystone-serdes-pcie";
> reg = <0x02320000 0x4000>;
> link-rate-kbps = <5000000>;
> num-lanes = <1>;
> status = "disabled";
> };
>
> pcie0: pcie@21800000 {
> compatible = "ti,keystone-pcie", "snps,dw-pcie";
> power-domains = <&k2g_pds K2G_DEV_PCIE0>;
> clocks = <&k2g_clks K2G_DEV_PCIE0
> K2G_DEV_PCIE_VBUS_CLK>;
> clock-names = "pcie";
> #address-cells = <3>;
> #size-cells = <2>;
> reg = <0x21801000 0x2000>, <0x21800000 0x1000>,
> <0x02620128 4>;
> ranges = <0x81000000 0 0 0x23250000 0 0x4000
> 0x82000000 0 0x50000000 0x50000000 0
> 0x10000000>;
>
> status = "disabled";
> device_type = "pci";
> num-lanes = <1>;
> phys = <&pcie0_phy>;
>
> #interrupt-cells = <1>;
> interrupt-map-mask = <0 0 0 7>;
> interrupt-map = <0 0 0 1 &pcie_intc0 0>, /* INT A */
> <0 0 0 2 &pcie_intc0 1>, /* INT B */
> <0 0 0 3 &pcie_intc0 2>, /* INT C */
> <0 0 0 4 &pcie_intc0 3>; /* INT D */
> pcie_msi_intc0: msi-interrupt-controller {
> interrupt-controller;
> #interrupt-cells = <1>;
> interrupt-parent = <&gic>;
> interrupts = <GIC_SPI 30
> IRQ_TYPE_EDGE_RISING>,
> <GIC_SPI 31 IRQ_TYPE_EDGE_RISING>,
> <GIC_SPI 32 IRQ_TYPE_EDGE_RISING>,
> <GIC_SPI 33 IRQ_TYPE_EDGE_RISING>,
> <GIC_SPI 34 IRQ_TYPE_EDGE_RISING>,
> <GIC_SPI 35 IRQ_TYPE_EDGE_RISING>,
> <GIC_SPI 36 IRQ_TYPE_EDGE_RISING>,
> <GIC_SPI 37 IRQ_TYPE_EDGE_RISING>;
> };
>
> pcie_intc0: legacy-interrupt-controller {
> interrupt-controller;
> #interrupt-cells = <1>;
> interrupt-parent = <&gic>;
> interrupts = <GIC_SPI 26
> IRQ_TYPE_EDGE_RISING>,
> <GIC_SPI 27 IRQ_TYPE_EDGE_RISING>,
> <GIC_SPI 28 IRQ_TYPE_EDGE_RISING>,
> <GIC_SPI 29 IRQ_TYPE_EDGE_RISING>;
> };
> };
>
>
>>
>>> However I can't get the Ethernet port functional.
>>
>>> Does this need MSI interrupt ?
>>
>> No.
>
> Does it also work with MSI?? Is there anything to be set in the driver
> to request MSI interrupt?
>
>>
>>> I can't see it has requested any.
>>
>> Yes, something went really, really wrong. See below.
>>
>> [...]
>>> [ 2.303965] PCI host bridge /soc/pcie@21800000 ranges:
>>> [ 2.309108] No bus range found for /soc/pcie@21800000, using [bus
>>> 00-ff]
>>> [ 2.316269] IO 0x23250000..0x23253fff -> 0x00000000
>>> [ 2.321499] MEM 0x50000000..0x5fffffff -> 0x50000000
>>> [ 2.331666] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
>>> [ 2.338283] pci_bus 0000:00: root bus resource [bus 00-ff]
>>> [ 2.343937] pci_bus 0000:00: root bus resource [io 0x0000-0x3fff]
>>> [ 2.350114] pci_bus 0000:00: root bus resource [mem
>>> 0x50000000-0x5fffffff]
>>> [ 2.357095] pci 0000:00:00.0: [104c:b00b] type 01 class 0x060400
>>> [ 2.357665] PCI: bus0: Fast back to back transfers disabled
>>> [ 2.363717] pci 0000:01:00.0: [10ec:8168] type 00 class 0x020000
>>> [ 2.363809] pci 0000:01:00.0: reg 0x10: [io 0x0000-0x00ff]
>>> [ 2.363867] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit]
>>> [ 2.363909] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x0000ffff 64bit
>>> pref]
>>> [ 2.363939] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0001ffff pref]
>>> [ 2.364099] pci 0000:01:00.0: supports D1 D2
>>> [ 2.364116] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
>>> [ 2.381251] PCI: bus1: Fast back to back transfers disabled
>>> [ 2.386989] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
>>> [ 2.393937] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff
>>> pref]
>>> [ 2.401221] pci 0000:00:00.0: BAR 7: assigned [io 0x1000-0x1fff]
>>> [ 2.407320] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff
>>> pref]
>>> [ 2.414597] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff
>>> 64bit pref]
>>> [ 2.422380] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff
>>> 64bit]
>>> [ 2.429702] pci 0000:01:00.0: BAR 0: assigned [io 0x1000-0x10ff]
>>> [ 2.435821] pci 0000:00:00.0: PCI bridge to [bus 01]
>>> [ 2.440783] pci 0000:00:00.0: bridge window [io 0x1000-0x1fff]
>>> [ 2.446896] pci 0000:00:00.0: bridge window [mem 0x50000000-0x500fffff]
>>> [ 2.453699] pci 0000:00:00.0: bridge window [mem 0x50100000-0x501fffff
>>> pref]
>>> [ 2.461453] pcieport 0000:00:00.0: Signaling PME through PCIe PME
>>> interrupt
>>> [ 2.468411] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
>>> [ 2.475075] pcie_pme 0000:00:00.0:pcie01: service driver pcie_pme loaded
>>> [ 2.475392] aer 0000:00:00.0:pcie02: service driver aer loaded
>>> [ 2.475652] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
>>> [ 2.481419] r8169 0000:01:00.0: enabling device (0140 -> 0143)
>>> [ 2.488865] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000,
>>> 00:00:00:00:00:00, XID 00000000 IRQ 286
>>
>> No need to go further, there is a serious problem here.
>>
>> Most of the XID bits are read from a mapped register. They're definitely not
>> expected to be null as the driver requires it to identify a proper chipset
>> (the common PCI configuration registers only tell that you aren't dealing
>> with a coffee machine).
>
> Looks like during enumeration, the PCIe controller has correctly read the
> Device ID a and Vendor ID over the PCIe bus. So the bus access is happening
> at least to read the common PCI configuration space. But then what you are
> confirming is the mapped XID register is not accessed correctly and is reading
> zeors. Right? I will try to debug this first before proceeding any further.
>
One difference between the BAR logs w.r.t Marvel controller on K2E EVM is that
the BARs are showing up as 64bit instead of 32bit.
Marvell controller
==================
[ 0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
[ 0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff
pref]
[ 0.236373] pci 0000:00:00.0: BAR 7: assigned [io 0x1000-0x1fff]
[ 0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff
pref]
[ 0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
[ 0.236406] pci 0000:01:00.0: BAR 4: assigned [io 0x1000-0x100f]
[ 0.236418] pci 0000:01:00.0: BAR 0: assigned [io 0x1010-0x1017]
[ 0.236429] pci 0000:01:00.0: BAR 2: assigned [io 0x1018-0x101f]
[ 0.236441] pci 0000:01:00.0: BAR 1: assigned [io 0x1020-0x1023]
[ 0.236452] pci 0000:01:00.0: BAR 3: assigned [io 0x1024-0x1027]
[ 0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 0.236472] pci 0000:00:00.0: bridge window [io 0x1000-0x1fff]
[ 0.236481] pci 0000:00:00.0: bridge window [mem 0x60000000-0x600fffff]
[ 0.236490] pci 0000:00:00.0: bridge window [mem 0x60100000-0x601fffff
pref]
Realtek
=========
[ 2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[ 2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 2.323844] pci_bus 0000:00: root bus resource [io 0x0000-0x3fff]
[ 2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
[ 2.337567] PCI: bus0: Fast back to back transfers disabled
[ 2.361159] PCI: bus1: Fast back to back transfers disabled
[ 2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
[ 2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff
pref]
[ 2.381061] pci 0000:00:00.0: BAR 7: assigned [io 0x1000-0x1fff]
[ 2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff
pref]
[ 2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff
64bit pref]
[ 2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff
64bit]
[ 2.409610] pci 0000:01:00.0: BAR 0: assigned [io 0x1000-0x10ff]
[ 2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 2.420693] pci 0000:00:00.0: bridge window [io 0x1000-0x1fff]
[ 2.426806] pci 0000:00:00.0: bridge window [mem 0x50000000-0x500fffff]
[ 2.433610] pci 0000:00:00.0: bridge window [mem 0x50100000-0x501fffff
pref]
Probably access to config space is happening in 64bit and it just get zeros?
Trying to see how does PCI core determine if the access is 64bit. If you have a
clue please let me know.
Murali
> Thanks.
>
> Murali
>
>> Any read returning zeroes would not surprize me.
>>
>> [...]
>>> Can someone help me figure out what is missing ?
>>
>> Hardly at this point. I can only suggest to switch power off, plug
>> the 8168 device in a x86 (32 or 64 bits) system and check how it behaves.
>>
> I need to continue work with my EVM as I have to make it functional.
>
>
--
Murali Karicheri
Linux Kernel, Keystone
