Am 03.12.2019 um 19:31 schrieb Ang, Chee Hong:
On Tue, Dec 3, 2019 at 3:45 PM Ang, Chee Hong <chee.hong....@intel.com> wrote:On Tue, Dec 3, 2019 at 2:37 AM Ang, Chee Hong <chee.hong....@intel.com> wrote:Am 02.12.2019 um 17:10 schrieb Ang, Chee Hong:On Mon, Dec 2, 2019 at 4:18 PM Ang, Chee Hong <chee.hong....@intel.com> wrote:On Mon, Dec 2, 2019 at 3:08 PM Ang, Chee Hong <chee.hong....@intel.com> wrote:On Mon, Dec 2, 2019 at 2:38 PM Ang, Chee Hong <chee.hong....@intel.com> wrote:On Mon, Dec 2, 2019 at 11:25 AM <chee.hong....@intel.com>wrote:From: "Ang, Chee Hong" <chee.hong....@intel.com> New U-boot flow with ARM Trusted Firmware (ATF) support: SPL (EL3) -> ATF-BL31 (EL3) -> U-Boot Proper (EL2) -> Linux (EL1)Adding support for ATF means that using U-Boot on Stratix10 and Agilex without ATF keeps working, right?ATF is needed in order for Stratix10 and Agilex's U-Boot to work.Is there a technical requirement for that?Yes. We are using ATF to provide PSCI services such as system reset (COLD reset), CPU_ON/CPU_OFF (CPU hotplug in Linux) and other secure services such as mailbox communications with Secure Device Manager and accessing the System Manager registers which aresecure.Without PSCI services, we are able to boot until U-Boot properonly.Currently, our U-Boot in mainline doesn't boot to Linux due to these missingPSCI services.Another reason is we have another boot flow which is using ATF +UEFI.So now we are re-using the PSCI services from ATF so that now U-Boot and UEFI share the same ATF-BL31 image so that we don't have toreimplement another sets of PSCI services for U-Boot again.This will greatly reduce our engineering efforts.Hmm, thanks for the explanation. I don't really think I can review this, given the lack of knowledge about that PSCI stuff.I believe you can review it. Have you briefly gone through the patches ? It has nothing to do with the PSCIstuff.It just call the invoke_smc() function to call the ATF's PSCI functions. Those PSCI functions in ATF will do the rest.No, not yet. But see below.I must say it seems strange to me that U-Boot would have to rely on ATFthough.How do other platforms implement this? Shouldn't PSCI be generic or is it really platform specific? If it's generic, isn't that a dupliation of code if you implement e.g. a reset driver for Stratix10 but callinto PSCI?It's not strange at all. A lot of U-Boot users already using this boot flow (ATF +U-Boot). Just because other boards do this doesn't mean it's not strange. Wasn't there some kind of discussion around that PSCI stuff to make itavailable from U-Boot?What's wrong with that way?Our downstream U-Boot branch is already implemented the PSCI stuffs in U-Boot.I tried to upstream my PSCI code: https://lists.denx.de/pipermail/u-boot/2019-May/368822.html Marek pointed me to this thread: https://www.mail-archive.com/u-boot@lists.denx.de/msg319458.ht ml He had a point. He suggested maybe we can implement the PSCI stuffs in SPL/TPL. I took a look at the U-Boot code and found out ATF is already wellsupported. Why don't we just use the PSCI code from ATF rather than re- implementing the PSCI code again in SPL/TPL.It will save our effort to maintain two PSCI code in U-Boot and ATF while wealready have the ATF PSCI implementation to support UEFI. It seems to me we do have working code in U-Boot, what's missing is "only" to turn it ino PSCI?Existing PSCI framework in U-Boot provide a way for us to turn the code into a PSCI handler by just adding a '__secure' keyword before thefunction name. See:https://gitlab.denx.de/u-boot/u-boot/blob/master/arch/arm/mach-soc fpga /mailbox_s10.c Below is one of the functions that has 2 versions. One 'live' in a normal code section and another one will be relocated to "__secure" section (for PSCI). You can see that 2 same functions are duplicated for normalcode section and PSCI section.int mbox_send_cmd(u8 id, u32 cmd, u8 is_indirect, u32 len, u32 *arg, u8 urgent, u32 *resp_buf_len, u32 *resp_buf) { return mbox_send_cmd_common(id, cmd, is_indirect, len, arg, urgent, resp_buf_len, resp_buf); } int __secure mbox_send_cmd_psci(u8 id, u32 cmd, u8 is_indirect, u32 len, u32 *arg, u8 urgent, u32 *resp_buf_len, u32 *resp_buf) { return mbox_send_cmd_common(id, cmd, is_indirect, len, arg, urgent, resp_buf_len, resp_buf); } Those functions that are needed by PSCI runtime need to be duplicated for"__secure" section.U-Boot Proper will copy and relocate the PSCI code in "__secure" section to a location before booting Linux whereby they can be called by Linux. Using the PSCI framework, U-Boot proper is not able to call any PSCIfunctions because PSCI code is not available until U-Boot proper ready to boot Linux.So that's the reason we need to have 2 sets of code in U-Boot. One for SPL/U-Boot and another one for PSCI section which is used by Linux. Currently we have 2 implementations for FPGA reconfiguration driver in ourdownstream branch.One for SPL/U-Boot and another one for Linux (PSCI). FPGA reconfiguration driver for U-Boot is already upstreamed but I don't think I canget the FPGA reconfiguration for the PSCI part upstreamed.They are 2 sets of different code for the same purpose. But that is what we have done in downstream to make sure we can support Linux. BTW, we are going to get rid of those duplicated code for PSCI after we switchto ATF boot flow. I think we have already discussed why that style is bad and unstable. The correct thing to do would be to compile an SPL style binary from the U-Boot sources that can replace ATF-BL31, not this messy __secure thing. I can see others (rockchip, TI, NXP?) might in part rely on ATF as well, but speaking for socfpga, if you must insist on using ATF, I would be happy if you could do it in a way that does not reduce existingfunctionality in U-Boot.Please do 'git grep CONFIG_SPL_ATF' and you will have some idea who are using the ATF with U-Boot. You can know which platforms are using the ATF by looking at the name of the defconfig files.That's where I found Rockchip, TI and NXP. I see I missed Xilinx.BTW, what makes you think this ATF method reduce the existing functionalityin U-Boot ?I don't really get that. I would like to know more to see what I can do to easeyour concern. You're making U-Boot (GPL-licensed) depend on ATF (BSD-licensed). That's my main concern. Then, you're making the whole build more complicated by having to build 2 independent projects (in matching versions, as they share at least one header file). I'd say it would be more straightforward to integrate PSCI services into U-Boot. I know that comes at the expense of someone taking the time to fix U-Boot PSCI support from "__secure" to a proper way. But I think the result would be cleaner. Added to that, with what you told me so far, you reduce U-Boot functionality by making the existing drivers in U-Boot proper require PSCI services, so U-Boot won't run standalone if you decide to not use ATF.We are trying to combine the best of both worlds (SPL/U-Boot + ATF) where we get to enjoy the existing benefits of using U-Boot and solid/well implemented PSCI services + ARMv8 extensions from ATF. Linux depends on those standard PSCI services such as CPU hotplug and etc and they are well implemented in ATF to avoid race conditions when waking/suspending the CPU cores. Besides, ATF also provide various ARMv8 extensions (such as RAS) to OS. So using ATF with U-Boot is much more than just PSCI. Beside the time/effort needed to fix the existing U-Boot PSCI issue, having a solid and well implemented PSCI services and ARMv8 extensions in U-Boot is another huge effort. We also need to ensure these implementations are well maintained and up to date with the PSCI/ARMv8 extensions specification from time to time. From U-Boot perspective, you have made some valid points. We understand using ATF with U-Boot do come with a cost. So far, the whole discussion only revolved around the issues/impact between ATF and U-Boot. Let's not forget the fact that ultimately most users will end up in OS environment (Linux) after SPL & U-Boot proper and this is what ATF do best by providing well implemented PSCI services and ARMv8 extensions to Linux. It might not be your concern.
As you might have noticed, I'm using gen5 with Linux at the moment and might at some point in the future be migrating to a 64-bit ARM, so that might well be S10/Agilex or some kind of stripped down version of that, you never know. So yes, in the future, this might be my concern :-)
But that's the benefits we see and I personally think this is one of the main reasons ATF support gets enabled in U-Boot.
I don't want to hold you back from using ATF + U-Boot. I only say I'm not convinced stripping down U-Boot functionality with that move is the way to go.
And I think it would be better to hard-code this at compile time (given the knowledge SPL -> raw, U-Boot -> via ATF) than evaluating EL.
Keeping that aside, what's the "secure" benefit of allowing register read/write/modify access via invoke_smc vs. directly modifying registers?
Regards, Simon
And given U-Boot aims to support UEFI (kind of?), I'd rather argue: why do you need ATF at all?No, U-Boot does not aim to support UEFI. We have 2 boot flows that don'tmix: Really? Or do you mean you don't aim to support EFI boot using U-Boot? I don't know that (U)EFI stuff too well, yet, but I was under the impression that Heinrich et. al. do want U-Boot to support UEFI?Yes. Currently, we have no plan to support (U)EFI boot with U-Boot. Anyway, I am not working on UEFI boot flow. That's the work from anotherteam.1) U-Boot -> ATF-BL31 -> U-Boot Proper -> Linux 2) ATF-BL2 -> ATF-BL31 -> UEFI -> Other OSes or Linux These two boot flows now share the same code base (ATF-BL31).Indeed, having the same code in both seems like double effort formaintenance.In my opinion, you're reducing functionality in U-Boot by making ATF a requirement. And by saying "I can't review this", I mean this looks like a questionable way to me and I'm not the one to say if a U-Boot board shouldgo this way or not.I understand. Btw, who should I include to review this ?Yes. PSCI is a generic software interface which encapsulate the platformspecific code.Let me give you a good example in one of your sysreset driver you haveimplemented for S10.#include <dm.h> #include <errno.h> #include <sysreset.h> -#include <asm/arch/mailbox_s10.h> static int socfpga_sysreset_request(struct udevice *dev, enum sysreset_t type) { - puts("Mailbox: Issuing mailbox cmd REBOOT_HPS\n"); - mbox_reset_cold(); + psci_system_reset();And coming back on this, the sysreset driver won't work in SPL any more,right?You brought a very good point. See my comment at the bottom.So this is not an socfgpa_s10 specific driver any more, right?This driver code can be renamed to more generic name such assocfpga_soc64.c.So that it can be shared by both Stratix10 and Agilex.return -EINPROGRESS; } Above is the changes in one of my patchsets, the sysreset driver for S10 used to call mbox_reset_cold() to send mailbox message to Secure Device Manager (SDM) to trigger COLD reset. Calling 'mbox_reset_cold()' means you are calling platform specific code in the sysreset driver to perform COLD reset. What if method to triggerCOLD reset is changed in new platform ?We have to change the sysreset driver code to support another newplatform.If this function call is replaced with "psci_system_reset()", we don't have to change the code at all because all the platform specific code for COLD reset is now reside in ATF because this function just invoke the PSCI function provided by ATF. You just have to replace the ATF binary with the new implementation for the new platform. We can re-use this sysreset driver for any platform that support ATF. In fact, it makes our U-Boot driver code more 'generic' because PSCI interface stay the same. BTW, Linux also call PSCI functions to perform COLD reset. Byusing ATF, U-Boot and Linux share the same COLD reset service provided byATF.It actually reduce code duplication. What I meant was code duplication inside the U-Boot tree (having one driver for each arch but in effect all those drivers will call into the same pscifunction).Can different archs share the same driver if the driver code is common tothose platforms ? I don't know why not. However, you then need a different way to select this driver: you clearly cannot use DT compatibles as this DT entry does not in any way stand for what you make the driver binding to itexecute.Instead, I would think of a way to make your PSCI-aware U-Boot proper use a generic PSCI-reset driver instead of the one matching the devicetree. And then keep in mind you still need the DT-matching driver in SPL. Thinking about it, having a driver in SPL you don't use in U-Boot proper is probably not done, yet, aswell.I don't have any problem with this approach (PSCI-reset driver) but it is very easy to support SPL and U-Boot proper in the same driver by justchecking the current exception level. Please take a look at the code below.#include <dm.h> #include <errno.h> #include <sysreset.h> #include <asm/arch/mailbox_s10.h> static int socfpga_sysreset_request(struct udevice *dev, enum sysreset_t type) { - puts("Mailbox: Issuing mailbox cmd REBOOT_HPS\n"); + If (current_el() == 3)Hard-coding the EL here seems quite a hack?+ mbox_reset_cold(); + else + psci_system_reset(); return -EINPROGRESS; } We can make the sysreset driver compatible in SPL and U-Boot proper by justchecking the current exception level.If it's EL3 (secure), we knew SPL is running and otherwise U-Boot proper (EL2,non-secure) is running. So you compile all the PSCI stuff into SPL although never using it?The PSCI stuff is just a very thin layer (interface to PSCI/SMC call) since realwork is done in ATF.Or we can do it in compile time: #ifdef CONFIG_SPL_BUILD // do it in normal way #else // invoke PSCI call #endifI'd still prefer to have DT-compat matched drivers implementing the hardware access. Then you can instantiate different drivers in U-Boot proper if you want to use PSCI, not the hardware. Having DT-compat matched drivers do something completely different (issuing PSCI calls instead of accessing the hardware they matched on) seems wrong.OK.Regards, SimonOr we can make a small generic function like below and call it from sysresetdriver code:void soc64_cold_reset(void) { If (current_el() == 3) mbox_reset_cold(); else psci_system_reset(); }What you're doing is to move this code from U-Boot open U-Boot sources to possibly closed source ATF sources. But we've already had that discussion, I think...Our PSCI implementation in ATF is open source: https://github.com/ARM-software/arm-trusted-firmware/tree/mast er/p lat/ intel/socWell, open source... Without implying anything: it's BSD, so it's open source as long as Intel wants it to be open source and nothing prevents the next manager from keeping additions or even bugfixes closedsource.For whatever reasons might come. Regards, SimonRegards, SimonI think you are aware of we are working to move the mailbox driver code awayfrom arch to drivers.You will see that a lot of those mailbox functions will be removed from themailbox driver.One of them is 'mbox_reset_cold()' which you called in sysreset driver forS10.Regards, SimonRegard, SimonSPL loads the u-boot.itb which consist of: 1) u-boot-nodtb.bin (U-Boot Proper image) 2) u-boot.dtb (U-Boot Proper DTB) 3) bl31.bin (ATF-BL31 image) Supported Platform: Intel SoCFPGA 64bits (Stratix10 & Agilex) Now, U-Boot Proper is running in non-secure mode (EL2), it invokes SMC/PSCI calls provided by ATF to perform COLD reset, System Manager register accesses and mailbox communications with Secure Device Manager(SDM).Steps to build the U-Boot with ATF support: 1) Build U-Boot 2) Build ATF BL31 3) Copy ATF BL31 binary image into U-Boot's root folder 4) "make u-boot.itb" to generate u-boot.itb These patchsets have dependency on: [U-Boot,v8,00/19] Add Intel Agilex SoC support: https://patchwork.ozlabs.org/cover/1201373/ Chee Hong Ang (19): arm: socfpga: add fit source file for pack itb with ATF arm: socfpga: Add function for checking description from FITimagearm: socfpga: Load FIT image with ATF support arm: socfpga: Override 'lowlevel_init' to support ATF configs: socfpga: Enable FIT image loading with ATF support arm: socfpga: Disable "spin-table" method for booting Linux arm: socfpga: Add SMC helper function for Intel SOCFPGA(64bits)arm: socfpga: Define SMC function identifiers for PSCI SiPservicesarm: socfpga: Add secure register access helper functions forSoC64bits arm: socfpga: Secure register access for clock manager (SoC64bits)arm: socfpga: Secure register access in PHY mode setup arm: socfpga: Secure register access for reading PLLfrequencymmc: dwmmc: socfpga: Secure register access in MMCdrivernet: designware: socfpga: Secure register access in MACdriverarm: socfpga: Secure register access in Reset Managerdriverarm: socfpga: stratix10: Initialize timer in SPL arm: socfpga: stratix10: Refactor FPGA reconfig driver to supportATFarm: socfpga: Bridge reset now invokes SMC calls to query FPGAconfigstatus sysreset: socfpga: Invoke PSCI call for COLD reset Dalon Westergreen (1): configs: stratix10: Remove CONFIG_OF_EMBEDThis one is included in another series already: https://patchwork.ozlabs.org/user/todo/uboot/?series=13 2976 Does this mean that one will be abandonen? So the combined hex output part of that series is not required anymore?Regards, Simonarch/arm/mach-socfpga/Kconfig | 2 - arch/arm/mach-socfpga/Makefile | 4 + arch/arm/mach-socfpga/board.c | 10 + arch/arm/mach-socfpga/clock_manager_agilex.c | 5 +- arch/arm/mach-socfpga/clock_manager_s10.c | 5 +- arch/arm/mach-socfpga/include/mach/misc.h | 3 + .../mach-socfpga/include/mach/secure_reg_helper.h | 20++arch/arm/mach-socfpga/lowlevel_init.S | 48 +++ arch/arm/mach-socfpga/misc_s10.c | 47 ++- arch/arm/mach-socfpga/reset_manager_s10.c | 31 +- arch/arm/mach-socfpga/secure_reg_helper.c | 67++++arch/arm/mach-socfpga/timer_s10.c | 3 +- arch/arm/mach-socfpga/wrap_pll_config_s10.c | 9 +- board/altera/soc64/its/fit_spl_atf.its | 51 +++ configs/socfpga_agilex_defconfig | 8 +- configs/socfpga_stratix10_defconfig | 9 +- drivers/fpga/stratix10.c | 261 ++++---------- drivers/mmc/socfpga_dw_mmc.c | 7 +- drivers/net/dwmac_socfpga.c | 5 +- drivers/sysreset/sysreset_socfpga_s10.c | 4 +- include/configs/socfpga_soc64_common.h | 2 +- include/linux/intel-smc.h | 374+++++++++++++++++++++22 files changed, 732 insertions(+), 243 deletions(-) create mode 100644 arch/arm/mach-socfpga/include/mach/secure_reg_helper.h create mode 100644 arch/arm/mach-socfpga/lowlevel_init.Screate mode 100644 arch/arm/mach-socfpga/secure_reg_helper.c create mode 100644 board/altera/soc64/its/fit_spl_atf.its create mode 100644 include/linux/intel-smc.h -- 2.7.4
