On Sat, 3 May 2025 00:07:56 -0400 Joel Fernandes <joelagn...@nvidia.com> wrote:
> Add explanation of fwsec with diagrams. This helps clarify how the > nova-core falcon boot works. > > Signed-off-by: Joel Fernandes <joelagn...@nvidia.com> > --- > Documentation/gpu/nova/core/fwsec.rst | 180 > ++++++++++++++++++++++++++ Documentation/gpu/nova/index.rst | > 1 + 2 files changed, 181 insertions(+) > create mode 100644 Documentation/gpu/nova/core/fwsec.rst > > diff --git a/Documentation/gpu/nova/core/fwsec.rst > b/Documentation/gpu/nova/core/fwsec.rst new file mode 100644 > index 000000000000..bed941ac3f2b > --- /dev/null > +++ b/Documentation/gpu/nova/core/fwsec.rst > @@ -0,0 +1,180 @@ > +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) > +========================= > +FWSEC (Firmware Security) > +========================= > +This document briefly/conceptually describes the FWSEC (Firmware > Security) image +and its role in the GPU boot sequence. As such, this > information is subject to +change in the future and is only current > as of the Ampere GPU family. However, +hopefully the concepts > described will be useful for understanding the kernel code +that > deals with it. All the information is derived from publicly available > +sources such as public drivers and documentation. + > +The role of FWSEC to provide secure boot, it is running in > Heavy-secure mode. It does +firmware verification after GPU reset and > load various ucode images on to the other +microcontrollers on the > GPU such as the PMU and GSP. + > +FWSEC itself is an application stored in the VBIOS ROM in the FWSEC > partition of +ROM (see vbios.rst for more details). It contains > different commands like FRTS +(Firmware Runtime Services) and SB > (Secure Booting other microcontrollers after +reset and loading them > with other non-FWSEC ucode). The kernel driver only needs to +to do > FRTS, since SB is already already after reset by the time the kernel > driver +is loaded. + > +The FRTS command carves out the WPR2 region (Write protected region) > which contains +data data required for power management. Once setup, repetitive word "data" ^ > only HS mode ucode can +access it (see falcon.rst for privilege > levels). + > +The FWSEC image is located in the VBIOS ROM in the partition of the > ROM that contains +various ucode images (also known as applications) > -- one of them being FWSEC. For how +it is extracted, see vbios.rst > and the vbios.rs source code. + > +The Falcon data for each ucode images (including the FWSEC image) is > a combination +of headers, data sections (DMEM) and instruction code > sections (IMEM). All these +ucode images are stored in the same ROM > partition and the PMU table is used to look +up the application to > load it based on its application ID (see vbios.rs). + > +For the purposes of nova-core driver, the FWSEC contains an > 'application interface' +called DMEMMAPPER which is used to the > "FWSEC-FRTS" command (among other commands it +is capable of > executing). For Ampere, FWSEC is running on the GSP in Heavy-secure > +mode and runs FRTS. + > +FWSEC Memory Layout > +------------------- > +The memory layout of the FWSEC image is as follows (this is using an > GA-102 +Ampere GPU as an example and could vary for future GPUs and > is subject to change +completely, it is just provided as an example): > + > +Here is a block diagram of the FWSEC memory layout:: > + ┌───────────────────────────────────────────────────────────────┐ > + │ FWSEC ROM image (type 0xE0) │ > + │ │ > + │ ┌─────────────────────────────────┐ │ > + │ │ PMU Falcon Ucode Table │ │ > + │ │ (PmuLookupTable) │ │ > + │ │ ┌─────────────────────────┐ │ │ > + │ │ │ Table Header │ │ │ > + │ │ │ - version: 0x01 │ │ │ > + │ │ │ - header_size: 6 │ │ │ > + │ │ │ - entry_size: 6 │ │ │ > + │ │ │ - entry_count: N │ │ │ > + │ │ │ - desc_version:3(unused)│ │ │ > + │ │ └─────────────────────────┘ │ │ > + │ │ ... │ │ > + │ │ ┌─────────────────────────┐ │ │ > + │ │ │ Entry for FWSEC (0x85) │ │ │ > + │ │ │ (PmuLookupTableEntry) │ │ │ > + │ │ │ - app_id: 0x85 (FWSEC) │ ───┼────┐ │ > + │ │ │ - target_id: 0x01 (PMU) │ │ │ │ > + │ │ │ - data: offset ─────────┼────┼────┼───┐ look up FWSEC │ > + │ │ └─────────────────────────┘ │ │ │ application. │ > + │ └─────────────────────────────────┘ │ │ │ > + │ │ │ │ > + │ │ │ │ > + │ ┌─────────────────────────────────┐ │ │ │ > + │ │ FWSEC Ucode Component │<───┘ │ │ > + │ │ (aka Falcon data) │ │ │ > + │ │ ┌─────────────────────────┐ │ │ │ > + │ │ │ FalconUCodeDescV3 │<───┼────────┘ │ > + │ │ │ - hdr │ │ │ > + │ │ │ - stored_size │ │ │ > + │ │ │ - pkc_data_offset │ │ │ > + │ │ │ - interface_offset ─────┼────┼────────────────┐ │ > + │ │ │ - imem_phys_base │ │ │ │ > + │ │ │ - imem_load_size │ │ │ │ > + │ │ │ - imem_virt_base │ │ │ │ > + │ │ │ - dmem_phys_base │ │ │ │ > + │ │ │ - dmem_load_size │ │ │ │ > + │ │ │ - engine_id_mask │ │ │ │ > + │ │ │ - ucode_id │ │ │ │ > + │ │ │ - signature_count │ │ look up sig │ │ > + │ │ │ - signature_versions --------------+ │ │ > + │ │ └─────────────────────────┘ │ │ │ │ > + │ │ (no gap) │ │ │ │ > + │ │ ┌─────────────────────────┐ │ │ │ │ > + │ │ │ Signatures Section │<───┼─────┘ │ │ > + │ │ │ (384 bytes per sig) │ │ │ │ > + │ │ │ - RSA-3K Signature 1 │ │ │ │ > + │ │ │ - RSA-3K Signature 2 │ │ │ │ > + │ │ │ ... │ │ │ │ > + │ │ └─────────────────────────┘ │ │ │ > + │ │ │ │ │ > + │ │ ┌─────────────────────────┐ │ │ │ > + │ │ │ IMEM Section (Code) │ │ │ │ > + │ │ │ │ │ │ │ > + │ │ │ Contains instruction │ │ │ │ > + │ │ │ code etc. │ │ │ │ > + │ │ └─────────────────────────┘ │ │ │ > + │ │ │ │ │ > + │ │ ┌─────────────────────────┐ │ │ │ > + │ │ │ DMEM Section (Data) │ │ │ │ > + │ │ │ │ │ │ │ > + │ │ │ ┌─────────────────────┐ │ │ │ │ > + │ │ │ │ Application │ │<───┼────────────────┘ │ > + │ │ │ │ Interface Table │ │ │ │ > + │ │ │ │ (FalconAppifHdrV1) │ │ │ │ > + │ │ │ │ Header: │ │ │ │ > + │ │ │ │ - version: 0x01 │ │ │ │ > + │ │ │ │ - header_size: 4 │ │ │ │ > + │ │ │ │ - entry_size: 8 │ │ │ │ > + │ │ │ │ - entry_count: N │ │ │ │ > + │ │ │ │ │ │ │ │ > + │ │ │ │ Entries: │ │ │ │ > + │ │ │ │ ┌─────────────────┐ │ │ │ │ > + │ │ │ │ │ DEVINIT (ID 1) │ │ │ │ │ > + │ │ │ │ │ - id: 0x01 │ │ │ │ │ > + │ │ │ │ │ - dmemOffset X ─┼─┼─┼────┐ │ > + │ │ │ │ └─────────────────┘ │ │ │ │ > + │ │ │ │ ┌─────────────────┐ │ │ │ │ > + │ │ │ │ │ DMEMMAPPER(ID 4)│ │ │ │ │ > + │ │ │ │ │ - id: 0x04 │ │ │ │ Used only for DevInit │ > + │ │ │ │ │ (NVFW_FALCON_ │ │ │ │ application (not FWSEC) │ > + │ │ │ │ │ APPIF_ID_DMEMMAPPER) │ │ > + │ │ │ │ │ - dmemOffset Y ─┼─┼─┼────┼─────┐ │ > + │ │ │ │ └─────────────────┘ │ │ │ │ │ > + │ │ │ └─────────────────────┘ │ │ │ │ > + │ │ │ │ │ │ │ > + │ │ │ ┌─────────────────────┐ │ │ │ │ > + │ │ │ │ DEVINIT Engine │<┼────┘ │ Used by FWSEC │ > + │ │ │ │ Interface │ │ │ │ app. │ > + │ │ │ └─────────────────────┘ │ │ │ │ > + │ │ │ │ │ │ │ > + │ │ │ ┌─────────────────────┐ │ │ │ │ > + │ │ │ │ DMEM Mapper (ID 4) │<┼────+─────┘ │ > + │ │ │ │ (FalconAppifDmemmapperV3) │ │ > + │ │ │ │ - signature: "DMAP" │ │ │ │ > + │ │ │ │ - version: 0x0003 │ │ │ │ > + │ │ │ │ - Size: 64 bytes │ │ │ │ > + │ │ │ │ - cmd_in_buffer_off │ │────┼────────────┐ │ > + │ │ │ │ - cmd_in_buffer_size│ │ │ │ │ > + │ │ │ │ - cmd_out_buffer_off│ │────┼────────────┼─────┐ │ > + │ │ │ │ - cmd_out_buffer_sz │ │ │ │ │ │ > + │ │ │ │ - init_cmd │ │ │ │ │ │ > + │ │ │ │ - features │ │ │ │ │ │ > + │ │ │ │ - cmd_mask0/1 │ │ │ │ │ │ > + │ │ │ └─────────────────────┘ │ │ │ │ │ > + │ │ │ │ │ │ │ │ > + │ │ │ ┌─────────────────────┐ │ │ │ │ │ > + │ │ │ │ Command Input Buffer│<┼────┼────────────┘ │ │ > + │ │ │ │ - Command data │ │ │ │ │ > + │ │ │ │ - Arguments │ │ │ │ │ > + │ │ │ └─────────────────────┘ │ │ │ │ > + │ │ │ │ │ │ │ > + │ │ │ ┌─────────────────────┐ │ │ │ │ > + │ │ │ │ Command Output │<┼────┼──────────────────┘ │ > + │ │ │ │ Buffer │ │ │ │ > + │ │ │ │ - Results │ │ │ │ > + │ │ │ │ - Status │ │ │ │ > + │ │ │ └─────────────────────┘ │ │ │ > + │ │ └─────────────────────────┘ │ │ > + │ └─────────────────────────────────┘ │ > + │ │ > + └───────────────────────────────────────────────────────────────┘ > + > +.. note:: > + The FWSEC image also plays a role in memory scrubbing (ECC > initialization) and VPR > + (Video Protected Region) initialization as well. Before the > nova-core driver is even > + loaded, the FWSEC image is running on the GSP in heavy-secure > mode. After the devinit > + sequence completes, it does VRAM memory scrubbing (ECC > initialization). On consumer > + GPUs, it scrubs only part of memory and then initiates 'async > scrubbing'. Before this > + async scrubbing completes, the unscrubbed VRAM cannot be used for > allocation (thus DRM > + memory allocators need to wait for this scrubbing to complete). > \ No newline at end of file > diff --git a/Documentation/gpu/nova/index.rst > b/Documentation/gpu/nova/index.rst index 91cc802ed94f..22e5712ac6b0 > 100644 --- a/Documentation/gpu/nova/index.rst > +++ b/Documentation/gpu/nova/index.rst > @@ -28,4 +28,5 @@ vGPU manager VFIO driver and the nova-drm driver. > > core/guidelines > core/vbios > + core/fwsec > core/todo
