Hi Pali, On Sun, Nov 7, 2021 at 4:58 PM Tony Dinh <mibo...@gmail.com> wrote: > > Hi Pali, > > Looks like it is working! But in a weird way. Please see my responses > in between, and also in the test log below. > > On Sun, Nov 7, 2021 at 3:45 PM Pali Rohár <p...@kernel.org> wrote: > > > > Hello! > > > > On Sunday 07 November 2021 12:56:37 Tony Dinh wrote: > > > Hi Pali, > > > > > > I've applied your patch, and ran some tests. Looks like we got to the > > > bind, but it was not probing (or the probe died silently). Later, I > > > tried pci enum and got some errors during probing. > > > > It looks like that arch/arm/mach-kirkwood/cpu.c already setup PCIe > > windows. Therefore when pci_mvebu.c tries it too, it fails on error > > "conflicts with another window" which you see in the log. > > > > Could you try to comment calling of "mvebu_mbus_add_window_by_id" > > function in pci_mvebu.c? As PCIe window setup should be done exactly > > once. > > > > Also try to call 'pci 0' and 'pci 1' after the 'pci enum'. > > > > > Here is the log (I added some debug printf to drivers/pci/pci_mvebu.c > > > and drivers/core/device.c to see the progress). > > > > > > - U-Boot boot log: > > > > > > U-Boot 2022.01-tld-1-00054-g52207514ba-dirty (Nov 06 2021 - 18:11:41 > > > -0700) > > > Pogoplug V4 > > > > > > SoC: Kirkwood 88F6281_A1 > > > DRAM: 128 MiB > > > mvebu_pcie_bind: begin > > > mvebu_pcie_bind: got an available node > > > Bound device to pcie@82000000 > > > mvebu_pcie_bind: bound > > > mvebu_pcie_bind: exit 0 > > > Bound device pcie@82000000 to mbus@f1000000 > > > Bound device mbus@f1000000 to root_driver > > > Bound device ehci@50000 to ocp@f1000000 > > > Bound device ethernet-controller@72000 to ocp@f1000000 > > > Bound device sata@80000 to ocp@f1000000 > > > Bound device mvs...@90000.blk to mvsdio@90000 > > > Bound device mvsdio@90000 to ocp@f1000000 > > > Bound device ocp@f1000000 to root_driver > > > NAND: 128 MiB > > > MMC: mvsdio@90000: 0 > > > Loading Environment from NAND... *** Warning - bad CRC, using default > > > environment > > > > > > In: serial > > > Out: serial > > > Err: serial > > > Net: > > > Warning: ethernet-controller@72000 (eth0) using random MAC address - > > > 66:43:1f:50:f0:e7 > > > eth0: ethernet-controller@72000 > > > Hit any key to stop autoboot: 0 > > > > > > Pogo_V4> pci > > > No such bus > > > Pogo_V4> pci 0 > > > No such bus > > > Pogo_V4> pci 1 > > > No such bus > > > > > > Pogo_V4> pci enum > > > mvebu_pcie_probe: > > > Cannot add window '4:e8', conflicts with another window > > > PCIe unable to add mbus window for mem at 90000000+10000000 > > > Cannot add window '4:e0', conflicts with another window > > > PCIe unable to add mbus window for IO at c0000000+00010000 > > > mvebu_pcie_probe: exit 0 > > > Bound device pci_0:0.0 to pcie0.0 > > > Bound device xhci_pci to pci_0:0.0 > > > > > > Pogo_V4> pci regions > > > # Bus start Phys start Size Flags > > > 0 0x0000000090000000 0x0000000090000000 0x0000000010000000 mem > > > 1 0x0000000000000000 0x0000000000000000 0x0000000008000000 mem sysmem > > > 2 0x00000000c0000000 0x00000000c0000000 0x0000000000010000 io io > > > > > > - Linux dmesg (grep for pci and xhci) > > > > > > [ 13.163011][ T1] mvebu-pcie mbus@f1000000:pcie@82000000: host > > > bridge /mbus@f1000000/pcie@82000000 ranges: > > > [ 13.163147][ T1] mvebu-pcie mbus@f1000000:pcie@82000000: > > > MEM 0x00f1040000..0x00f1041fff -> 0x0000040000 > > > [ 13.163217][ T1] mvebu-pcie mbus@f1000000:pcie@82000000: > > > MEM 0xffffffffffffffff..0x00fffffffe -> 0x0100000000 > > > [ 13.163268][ T1] mvebu-pcie mbus@f1000000:pcie@82000000: > > > IO 0xffffffffffffffff..0x00fffffffe -> 0x0100000000 > > > [ 13.163754][ T1] mvebu-pcie mbus@f1000000:pcie@82000000: PCI > > > host bridge to bus 0000:00 > > > [ 13.163791][ T1] pci_bus 0000:00: root bus resource [bus 00-ff] > > > [ 13.163829][ T1] pci_bus 0000:00: root bus resource [mem > > > 0xf1040000-0xf1041fff] (bus address [0x00040000-0x00041fff]) > > > [ 13.163862][ T1] pci_bus 0000:00: root bus resource [mem > > > 0xe0000000-0xefffffff] > > > [ 13.163892][ T1] pci_bus 0000:00: root bus resource [io > > > 0x1000-0xeffff] > > > [ 13.164075][ T1] pci 0000:00:01.0: [11ab:6281] type 01 class > > > 0x060400 > > > [ 13.164133][ T1] pci 0000:00:01.0: reg 0x38: [mem > > > 0x00000000-0x000007ff pref] > > > [ 13.165893][ T1] PCI: bus0: Fast back to back transfers disabled > > > [ 13.165949][ T1] pci 0000:00:01.0: bridge configuration invalid > > > ([bus 00-00]), reconfiguring > > > [ 13.166276][ T1] pci 0000:01:00.0: [1b73:1009] type 00 class > > > 0x0c0330 > > > [ 13.166334][ T1] pci 0000:01:00.0: reg 0x10: [mem > > > 0x00000000-0x0000ffff 64bit] > > > [ 13.166381][ T1] pci 0000:01:00.0: reg 0x18: [mem > > > 0x00000000-0x00000fff 64bit] > > > [ 13.166424][ T1] pci 0000:01:00.0: reg 0x20: [mem > > > 0x00000000-0x00000fff 64bit] > > > [ 13.166561][ T1] pci 0000:01:00.0: supports D1 > > > [ 13.166588][ T1] pci 0000:01:00.0: PME# supported from D0 D1 D3hot > > > [ 13.166685][ T1] pci 0000:01:00.0: 2.000 Gb/s available PCIe > > > bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:00:01.0 (capable > > > of 4.000 Gb/s with 5.0 GT/s PCIe x1 link) > > > [ 13.197158][ T1] PCI: bus1: Fast back to back transfers disabled > > > [ 13.197208][ T1] pci_bus 0000:01: busn_res: [bus 01-ff] end is > > > updated to 01 > > > [ 13.197538][ T1] pci 0000:00:01.0: BAR 14: assigned [mem > > > 0xe0000000-0xe00fffff] > > > [ 13.197581][ T1] pci 0000:00:01.0: BAR 6: assigned [mem > > > 0xe0100000-0xe01007ff pref] > > > [ 13.197624][ T1] pci 0000:01:00.0: BAR 0: assigned [mem > > > 0xe0000000-0xe000ffff 64bit] > > > [ 13.197670][ T1] pci 0000:01:00.0: BAR 2: assigned [mem > > > 0xe0010000-0xe0010fff 64bit] > > > [ 13.197715][ T1] pci 0000:01:00.0: BAR 4: assigned [mem > > > 0xe0011000-0xe0011fff 64bit] > > > [ 13.197759][ T1] pci 0000:00:01.0: PCI bridge to [bus 01] > > > [ 13.197792][ T1] pci 0000:00:01.0: bridge window [mem > > > 0xe0000000-0xe00fffff] > > > [ 13.197985][ T1] pcieport 0000:00:01.0: enabling device (0140 -> > > > 0142) > > > [ 13.198135][ T1] pci 0000:01:00.0: enabling device (0140 -> 0142) > > > > > > [ 15.619756][ T1] xhci_hcd 0000:01:00.0: xHCI Host Controller > > > [ 15.625694][ T1] xhci_hcd 0000:01:00.0: new USB bus registered, > > > assigned bus number 2 > > > [ 15.635643][ T1] xhci_hcd 0000:01:00.0: hcc params 0x200073a1 > > > hci version 0x100 quirks 0x0000000000080010 > > > [ 15.646477][ T1] xhci_hcd 0000:01:00.0: xHCI Host Controller > > > [ 15.652432][ T1] xhci_hcd 0000:01:00.0: new USB bus registered, > > > assigned bus number 3 > > > [ 15.660560][ T1] xhci_hcd 0000:01:00.0: Host supports USB 3.0 > > > SuperSpeed > > > > > > Note that this board PCI and USB 3.0 have been working in the Linux > > > kernel for quite many years. > > > > > > - These are relevant configs used in the build (not sure if I need to > > > use CONFIG_DM_PCI_COMPAT?) > > > > > > CONFIG_DM=y > > > CONFIG_CMD_PCI=y > > > CONFIG_PCI=y > > > CONFIG_PCI_MVEBU=y > > > CONFIG_PCI_ENDPOINT=y > > > CONFIG_DM_PCI_COMPAT=y > > > CONFIG_USB_XHCI_HCD=y > > > CONFIG_USB_XHCI_PCI=y > > > CONFIG_USB_XHCI_MVEBU=y > > > > Following options should be enough: > > > > CONFIG_CMD_PCI=y > > CONFIG_PCI=y > > # CONFIG_DM_PCI_COMPAT is not set > > CONFIG_PCI_PNP=y > > CONFIG_PCI_MVEBU=y > > Since the PCI bus in this box is used for USB 3.0, I've included > various XHCI configs to see if I can probe this USB 3.0 flash drive > also (that's the ultimate test). > > > > > Option CONFIG_PCI_PNP=y is required to see endpoint cards. Without it > > U-Boot would see only PCIe Root Ports. But for first basic tests it > > should be enough. > > > > > BTW, the topic is no longer kwboot, should we move this to another new > > > thread? i.e. Testing PCI MVEBU with Kirkwood SoCs. > > > > Changed :-) > > Thanks :) > > Here is the log of a successful spin up for the PCI bus and USB 3.0 > drive. Please see my commentary ### in between the logs. > > U-Boot 2022.01-rc1-00054-g52207514ba-dirty (Nov 07 2021 - 16:03:46 -0800) > Pogoplug V4 > > SoC: Kirkwood 88F6281_A1 > DRAM: 128 MiB > device_probe: > device_probe: exit 0 > > > ### The device_probe above seems to have nothing happening. > > > mvebu_pcie_bind: begin > mvebu_pcie_bind: got an available node > Bound device to pcie@82000000 > mvebu_pcie_bind: bound > mvebu_pcie_bind: exit 0 > Bound device pcie@82000000 to mbus@f1000000 > Bound device mbus@f1000000 to root_driver > Bound device ehci@50000 to ocp@f1000000 > Bound device ethernet-controller@72000 to ocp@f1000000 > Bound device sata@80000 to ocp@f1000000 > Bound device mvs...@90000.blk to mvsdio@90000 > Bound device mvsdio@90000 to ocp@f1000000 > Bound device ocp@f1000000 to root_driver > NAND: 128 MiB > MMC: device_probe: > device_probe: > device_probe: > device_probe: exit 0 > device_probe: exit 0 > device_probe: > device_probe: > mvsdio@90000: 0 > Loading Environment from NAND... *** Warning - bad CRC, using default > environment > > In: serial > Out: serial > Err: serial > Net: device_probe: > device_probe: > > Warning: ethernet-controller@72000 (eth0) using random MAC address - > 86:b8:84:ee:a9:a5 > device_probe: exit 0 > eth0: ethernet-controller@72000 > Hit any key to stop autoboot: 0 > Pogo_V4> usb reset > resetting USB... > Bus ehci@50000: device_probe: > device_probe: > USB EHCI 1.00 > device_probe: exit 0 > scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000 > device_probe: > device_probe: > Bound device usb_hub to usb_hub > device_probe: > device_probe: > device_probe: exit 0 > Bound device usb_mass_storage to usb_hub > device_probe: > device_probe: > Bound device usb_mass_storage.lun0 to usb_mass_storage > device_probe: exit 0 > device_probe: exit 0 > 3 USB Device(s) found > scanning usb for storage devices... 1 Storage Device(s) found > > > ### Try to bring up both USB drives (2.0 and 3.0). But only the USB > 2.0 was up as shown above. > > > Pogo_V4> pci enum > device_probe: > device_probe: > device_probe: > device_probe: > device_probe: exit 0 > device_probe: exit 0 > mvebu_pcie_probe: > Cannot add window '4:e8', conflicts with another window > PCIe unable to add mbus window for mem at 90000000+10000000 > Cannot add window '4:e0', conflicts with another window > PCIe unable to add mbus window for IO at c0000000+00010000 > mvebu_pcie_probe: exit 0 > Bound device pci_0:0.0 to pcie0.0 > device_probe: > device_probe: > Bound device xhci_pci to pci_0:0.0 > device_probe: exit 0 > device_probe: exit 0 > device_probe: > > ### So I did the pci enum above to get the XHCI bound to PCIe. > > Pogo_V4> usb reset > resetting USB... > Bus ehci@50000: device_probe: > device_probe: > USB EHCI 1.00 > device_probe: exit 0 > Bus xhci_pci: device_probe: > device_probe: > Register 400081f NbrPorts 4 > Starting the controller > USB XHCI 1.00 > device_probe: exit 0 > scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000 > device_probe: > device_probe: > Bound device usb_hub to usb_hub > device_probe: > device_probe: > device_probe: exit 0 > Bound device usb_mass_storage to usb_hub > device_probe: > device_probe: > Bound device usb_mass_storage.lun0 to usb_mass_storage > device_probe: exit 0 > device_probe: exit 0 > 3 USB Device(s) found > scanning bus xhci_pci for devices... Bound device usb_hub to xhci_pci > device_probe: > device_probe: > Bound device usb_mass_storage to usb_hub > device_probe: > device_probe: > Bound device usb_mass_storage.lun0 to usb_mass_storage > device_probe: exit 0 > device_probe: exit 0 > 2 USB Device(s) found > scanning usb for storage devices... 2 Storage Device(s) found > > > ### Now both drives are up! > > > Pogo_V4> pci 0 > device_probe: > Scanning PCI devices on bus 0 > BusDevFun VendorId DeviceId Device Class Sub-Class > _____________________________________________________________ > 00.00.00 0x11ab 0x6281 Bridge device 0x04 > Pogo_V4> pci 1 > device_probe: > Scanning PCI devices on bus 1 > BusDevFun VendorId DeviceId Device Class Sub-Class > _____________________________________________________________ > 01.00.00 0x1b73 0x1009 Serial bus controller 0x03 > > > ### And the pci command has meaningful output at this point as shown above. > > > Pogo_V4> usb tree > USB device tree: > 1 Hub (480 Mb/s, 0mA) > | u-boot EHCI Host Controller > | > +-2 Hub (480 Mb/s, 100mA) > | GenesysLogic USB2.0 Hub > | > +-3 Mass Storage (480 Mb/s, 224mA) > SanDisk Ultra Fit 4C531001560827107320 > > 1 Hub (5 Gb/s, 0mA) > | U-Boot XHCI Host Controller > | > +-2 Mass Storage (5 Gb/s, 224mA) > SanDisk Cruzer Glide 3.0 4C530000130418116112 > > Pogo_V4> usb info > 1: Hub, USB Revision 2.0 > - u-boot EHCI Host Controller > - Class: Hub > - PacketSize: 64 Configurations: 1 > - Vendor: 0x0000 Product 0x0000 Version 1.0 > Configuration: 1 > - Interfaces: 1 Self Powered 0mA > Interface: 0 > - Alternate Setting 0, Endpoints: 1 > - Class Hub > - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms > > 2: Hub, USB Revision 2.0 > - GenesysLogic USB2.0 Hub > - Class: Hub > - PacketSize: 64 Configurations: 1 > - Vendor: 0x05e3 Product 0x0610 Version 144.48 > Configuration: 1 > - Interfaces: 1 Self Powered Remote Wakeup 100mA > Interface: 0 > - Alternate Setting 0, Endpoints: 1 > - Class Hub > - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms > - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms > > 3: Mass Storage, USB Revision 2.10 > - SanDisk Ultra Fit 4C531001560827107320 > - Class: (from Interface) Mass Storage > - PacketSize: 64 Configurations: 1 > - Vendor: 0x0781 Product 0x5583 Version 1.0 > Configuration: 1 > - Interfaces: 1 Bus Powered 224mA > Interface: 0 > - Alternate Setting 0, Endpoints: 2 > - Class Mass Storage, Transp. SCSI, Bulk only > - Endpoint 1 In Bulk MaxPacket 512 > - Endpoint 2 Out Bulk MaxPacket 512 > > 1: Hub, USB Revision 3.0 > - U-Boot XHCI Host Controller > - Class: Hub > - PacketSize: 512 Configurations: 1 > - Vendor: 0x0000 Product 0x0000 Version 1.0 > Configuration: 1 > - Interfaces: 1 Self Powered 0mA > Interface: 0 > - Alternate Setting 0, Endpoints: 1 > - Class Hub > - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms > > 2: Mass Storage, USB Revision 3.0 > - SanDisk Cruzer Glide 3.0 4C530000130418116112 > - Class: (from Interface) Mass Storage > - PacketSize: 512 Configurations: 1 > - Vendor: 0x0781 Product 0x5597 Version 1.0 > Configuration: 1 > - Interfaces: 1 Bus Powered 224mA > Interface: 0 > - Alternate Setting 0, Endpoints: 2 > - Class Mass Storage, Transp. SCSI, Bulk only > - Endpoint 1 In Bulk MaxPacket 1024 > - Endpoint 2 Out Bulk MaxPacket 1024 > > ### The XHCI Host Controller and the SanDisk Cruzer Glide are now > active as shown above. > > ### So list the files on this SanDisk Cruzer Glide to make sure we can read > it. > > > Pogo_V4> ext2ls usb 1:1 / > device_probe: > device_probe: > device_probe: exit 0 > <DIR> 4096 . > <DIR> 4096 .. > 522666 uboot.2021.07-tld-1.pogo_v4.bin > 524288 uboot.2021.07-tld-1.pogo_v4.mtd0.kwb > > Great works Pali! > > I think some more investigation is needed. Why did we need to do "pci > enum", and then "usb reset", in that order, to get the PCI bus and the > XHCI controller probed? there must be something missing in the process > somewhere between the Device uclass, the PCI uclass, and the pci_mvebu > uclass?
I think I can see the order of enumeration. PCI must be enumerated first, and then XHCI being the controller on this host bus would come alive? I think we can live with 'pci enum' and 'usb reset' to get the USB 3.0 drives enumerated. However, it seems just a little bit unintuitive. Thanks, Tony
