Hi, There are two things here. 1. System fails to boot i.e., hangs 2. FPGA's connected to root port are not detected in FW/OS
For problem 1, can you give data on where exactly it hangs?, Is it in OS or FW ?, Can you provide kernel/coreboot log, port 80 dump when it hangs. For problem 2, Can you try setting UPD PcieRpHotPlug, & check the behaviour. reference: https://review.coreboot.org/cgit/coreboot.git/tree/src/soc/intel/skylake/chip_fsp20.c#n300 Can you please check FPGA datasheet & check for frequency tolerance of PCIE_CLK_REF signal from FPGA. Also can you re-verify board layout for any PDG(Platform Design Guide) violations like impedance, length matching, limits for differential PCIE_CLK_REF, PCIe Lanes. Sometimes noise generated within board can generate huge EMI. I assume high noisy circuits(power supply, backlight driver etc) are kept away from high speed PCIe signals. Regards, Naresh G Solanki On Mon, Oct 15, 2018 at 5:37 PM Christian Gmeiner <[email protected]> wrote: > > Am Fr., 12. Okt. 2018 um 10:15 Uhr schrieb Nico Huber <[email protected]>: > > > > On 10/11/18 11:29 AM, Christian Gmeiner wrote: > > > During the last weeks I found the root cause of my problem - PCIe > > > spread spectrum > > > > > > Our FPGAs need a stable 100MHz PCIE clock to work. The used FSP config > > > thing looked > > > like this: > > > > > > void mainboard_memory_init_params(FSPM_UPD *mupd) > > > { > > > FSP_M_CONFIG *mem_cfg; > > > struct spd_block blk = { > > > .addr_map = { 0x50 }, > > > }; > > > > > > mem_cfg = &mupd->FspmConfig; > > > > > > mem_cfg->PegDisableSpreadSpectrumClocking = 1; > > > mem_cfg->PchPmPciePllSsc = 0; > > > > > > ... > > > } > > > > > > With this configuration the PCIe reference clock was off more then 8% > > > which > > > caused the system to hang during cold and warm boots. > > > > > > In the next step I removed assignment of PchPmPciePllSsc as it is > > > documented > > > as 'No BIOS override'. With this change I got more then 1000 soft and > > > 2000 hard reboots > > > without any problem. Keep in mind we started with only 10 successful > > > reboots. > > > > Please be more specific about the final setting of this UPD. `No BIOS > > override` is the documentation for the default value of 0xff. But is > > this set to the default in the binary? who knows... > > > > void mainboard_memory_init_params(FSPM_UPD *mupd) > { > FSP_M_CONFIG *mem_cfg; > struct spd_block blk = { > .addr_map = { 0x50 }, > }; > > mem_cfg = &mupd->FspmConfig; > > /* Disable PCIe Spread Spectrum Clocking */ > printk(BIOS_ERR, "PegDisableSpreadSpectrumClocking: %x\n", > mem_cfg->PegDisableSpreadSpectrumClocking); > printk(BIOS_ERR, "PchPmPciePllSsc: %x\n", mem_cfg->PchPmPciePllSsc); > mem_cfg->PegDisableSpreadSpectrumClocking = 1; > > get_spd_smbus(&blk); > dump_spd_info(&blk); > assert(blk.spd_array[0][0] != 0); > > mainboard_fill_dq_map_data(&mem_cfg->DqByteMapCh0); > mainboard_fill_dqs_map_data(&mem_cfg->DqsMapCpu2DramCh0); > mainboard_fill_rcomp_res_data(&mem_cfg->RcompResistor); > mainboard_fill_rcomp_strength_data(&mem_cfg->RcompTarget); > > mem_cfg->DqPinsInterleaved = TRUE; > mem_cfg->MemorySpdDataLen = blk.len; > mem_cfg->MemorySpdPtr00 = (uintptr_t) blk.spd_array[0]; > } > > And here is the output taken from cbmem -1: > > .. > FMAP: base = ff000000 size = 1000000 #areas = 4 > FMAP: area RW_MRC_CACHE found @ a50000 (65536 bytes) > MRC: no data in 'RW_MRC_CACHE' > PegDisableSpreadSpectrumClocking: 0 > PchPmPciePllSsc: ff > SPD @ 0x50 > SPD: module type is DDR4 > .. > > > > > > > The big problem is that PegDisableSpreadSpectrumClocking has no effect > > > at all. I measured > > > the freq it is not the 100MHz as expected. And I need to have a stable > > > 100MHz this clock source > > > is used internally by the FPGA to drive internal clocks. The end > > > results is that EtherCAT is not > > > able to sync. > > > > This setting is about a different clock, I guess. Can you please clarify > > what is connected to which clock on your board. > > > > Our FPGAs are using the 100 MHz PCIe clock as input to drive internal > clocks etc. One > of these clock is used for EtherCAT. If spread spectrum is active we > are around 100 MHz > (worst measured freq was ~92 MHz) and as a result the internal FPGA clock is > not > stable/reliable --> EtherCAT sync fails. > > If I use the following pattern: > mem_cfg->PegDisableSpreadSpectrumClocking = 1; > mem_cfg->PchPmPciePllSsc = 0; > > I get a stable 100 MHz PCIe clock signal and everything works - except > the device hangs > after < 10 warm reboots. Looks like PCIe link training fails > uncountable times (seen with protocol > analyzer). > > -- > greets > -- > Christian Gmeiner, MSc > > https://christian-gmeiner.info > > -- > coreboot mailing list: [email protected] > https://mail.coreboot.org/mailman/listinfo/coreboot -- Best regards, Naresh G. Solanki -- coreboot mailing list: [email protected] https://mail.coreboot.org/mailman/listinfo/coreboot
