Add explanation of fwsec with diagrams. This helps clarify how the
nova-core falcon boot works.
Signed-off-by: Joel Fernandes <joelagn...@nvidia.com>
---
Documentation/gpu/nova/core/fwsec.rst | 180 ++++++++++++++++++++++++++
Documentation/gpu/nova/index.rst | 1 +
2 files changed, 181 insertions(+)
create mode 100644 Documentation/gpu/nova/core/fwsec.rst
diff --git a/Documentation/gpu/nova/core/fwsec.rst
b/Documentation/gpu/nova/core/fwsec.rst
new file mode 100644
index 000000000000..bed941ac3f2b
--- /dev/null
+++ b/Documentation/gpu/nova/core/fwsec.rst
@@ -0,0 +1,180 @@
+.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+=========================
+FWSEC (Firmware Security)
+=========================
+This document briefly/conceptually describes the FWSEC (Firmware Security)
image
+and its role in the GPU boot sequence. As such, this information is subject to
+change in the future and is only current as of the Ampere GPU family. However,
+hopefully the concepts described will be useful for understanding the kernel
code
+that deals with it. All the information is derived from publicly available
+sources such as public drivers and documentation.
+
+The role of FWSEC to provide secure boot, it is running in Heavy-secure mode.
It does
+firmware verification after GPU reset and load various ucode images on to the
other
+microcontrollers on the GPU such as the PMU and GSP.
+
+FWSEC itself is an application stored in the VBIOS ROM in the FWSEC partition
of
+ROM (see vbios.rst for more details). It contains different commands like FRTS
+(Firmware Runtime Services) and SB (Secure Booting other microcontrollers after
+reset and loading them with other non-FWSEC ucode). The kernel driver only
needs to
+to do FRTS, since SB is already already after reset by the time the kernel
driver
+is loaded.
+
+The FRTS command carves out the WPR2 region (Write protected region) which
contains
+data data required for power management. Once setup, only HS mode ucode can
+access it (see falcon.rst for privilege levels).
+
+The FWSEC image is located in the VBIOS ROM in the partition of the ROM that
contains
+various ucode images (also known as applications) -- one of them being FWSEC.
For how
+it is extracted, see vbios.rst and the vbios.rs source code.
+
+The Falcon data for each ucode images (including the FWSEC image) is a
combination
+of headers, data sections (DMEM) and instruction code sections (IMEM). All
these
+ucode images are stored in the same ROM partition and the PMU table is used to
look
+up the application to load it based on its application ID (see vbios.rs).
+
+For the purposes of nova-core driver, the FWSEC contains an 'application
interface'
+called DMEMMAPPER which is used to the "FWSEC-FRTS" command (among other
commands it
+is capable of executing). For Ampere, FWSEC is running on the GSP in
Heavy-secure
+mode and runs FRTS.
+
+FWSEC Memory Layout
+-------------------
+The memory layout of the FWSEC image is as follows (this is using an GA-102
+Ampere GPU as an example and could vary for future GPUs and is subject to
change
+completely, it is just provided as an example):
+
+Here is a block diagram of the FWSEC memory layout::
+ ┌───────────────────────────────────────────────────────────────┐
+ │ FWSEC ROM image (type 0xE0) │
+ │ │
+ │ ┌─────────────────────────────────┐ │
+ │ │ PMU Falcon Ucode Table │ │
+ │ │ (PmuLookupTable) │ │
+ │ │ ┌─────────────────────────┐ │ │
+ │ │ │ Table Header │ │ │
+ │ │ │ - version: 0x01 │ │ │
+ │ │ │ - header_size: 6 │ │ │
+ │ │ │ - entry_size: 6 │ │ │
+ │ │ │ - entry_count: N │ │ │
+ │ │ │ - desc_version:3(unused)│ │ │
+ │ │ └─────────────────────────┘ │ │
+ │ │ ... │ │
+ │ │ ┌─────────────────────────┐ │ │
+ │ │ │ Entry for FWSEC (0x85) │ │ │
+ │ │ │ (PmuLookupTableEntry) │ │ │
+ │ │ │ - app_id: 0x85 (FWSEC) │ ───┼────┐ │
+ │ │ │ - target_id: 0x01 (PMU) │ │ │ │
+ │ │ │ - data: offset ─────────┼────┼────┼───┐ look up FWSEC │
+ │ │ └─────────────────────────┘ │ │ │ application. │
+ │ └─────────────────────────────────┘ │ │ │
+ │ │ │ │
+ │ │ │ │
+ │ ┌─────────────────────────────────┐ │ │ │
+ │ │ FWSEC Ucode Component │<───┘ │ │
+ │ │ (aka Falcon data) │ │ │
+ │ │ ┌─────────────────────────┐ │ │ │
+ │ │ │ FalconUCodeDescV3 │<───┼────────┘ │
+ │ │ │ - hdr │ │ │
+ │ │ │ - stored_size │ │ │
+ │ │ │ - pkc_data_offset │ │ │
+ │ │ │ - interface_offset ─────┼────┼────────────────┐ │
+ │ │ │ - imem_phys_base │ │ │ │
+ │ │ │ - imem_load_size │ │ │ │
+ │ │ │ - imem_virt_base │ │ │ │
+ │ │ │ - dmem_phys_base │ │ │ │
+ │ │ │ - dmem_load_size │ │ │ │
+ │ │ │ - engine_id_mask │ │ │ │
+ │ │ │ - ucode_id │ │ │ │
+ │ │ │ - signature_count │ │ look up sig │ │
+ │ │ │ - signature_versions --------------+ │ │
+ │ │ └─────────────────────────┘ │ │ │ │
+ │ │ (no gap) │ │ │ │
+ │ │ ┌─────────────────────────┐ │ │ │ │
+ │ │ │ Signatures Section │<───┼─────┘ │ │
+ │ │ │ (384 bytes per sig) │ │ │ │
+ │ │ │ - RSA-3K Signature 1 │ │ │ │
+ │ │ │ - RSA-3K Signature 2 │ │ │ │
+ │ │ │ ... │ │ │ │
+ │ │ └─────────────────────────┘ │ │ │
+ │ │ │ │ │
+ │ │ ┌─────────────────────────┐ │ │ │
+ │ │ │ IMEM Section (Code) │ │ │ │
+ │ │ │ │ │ │ │
+ │ │ │ Contains instruction │ │ │ │
+ │ │ │ code etc. │ │ │ │
+ │ │ └─────────────────────────┘ │ │ │
+ │ │ │ │ │
+ │ │ ┌─────────────────────────┐ │ │ │
+ │ │ │ DMEM Section (Data) │ │ │ │
+ │ │ │ │ │ │ │
+ │ │ │ ┌─────────────────────┐ │ │ │ │
+ │ │ │ │ Application │ │<───┼────────────────┘ │
+ │ │ │ │ Interface Table │ │ │ │
+ │ │ │ │ (FalconAppifHdrV1) │ │ │ │
+ │ │ │ │ Header: │ │ │ │
+ │ │ │ │ - version: 0x01 │ │ │ │
+ │ │ │ │ - header_size: 4 │ │ │ │
+ │ │ │ │ - entry_size: 8 │ │ │ │
+ │ │ │ │ - entry_count: N │ │ │ │
+ │ │ │ │ │ │ │ │
+ │ │ │ │ Entries: │ │ │ │
+ │ │ │ │ ┌─────────────────┐ │ │ │ │
+ │ │ │ │ │ DEVINIT (ID 1) │ │ │ │ │
+ │ │ │ │ │ - id: 0x01 │ │ │ │ │
+ │ │ │ │ │ - dmemOffset X ─┼─┼─┼────┐ │
+ │ │ │ │ └─────────────────┘ │ │ │ │
+ │ │ │ │ ┌─────────────────┐ │ │ │ │
+ │ │ │ │ │ DMEMMAPPER(ID 4)│ │ │ │ │
+ │ │ │ │ │ - id: 0x04 │ │ │ │ Used only for DevInit │
+ │ │ │ │ │ (NVFW_FALCON_ │ │ │ │ application (not FWSEC) │
+ │ │ │ │ │ APPIF_ID_DMEMMAPPER) │ │
+ │ │ │ │ │ - dmemOffset Y ─┼─┼─┼────┼─────┐ │
+ │ │ │ │ └─────────────────┘ │ │ │ │ │
+ │ │ │ └─────────────────────┘ │ │ │ │
+ │ │ │ │ │ │ │
+ │ │ │ ┌─────────────────────┐ │ │ │ │
+ │ │ │ │ DEVINIT Engine │<┼────┘ │ Used by FWSEC │
+ │ │ │ │ Interface │ │ │ │ app. │
+ │ │ │ └─────────────────────┘ │ │ │ │
+ │ │ │ │ │ │ │
+ │ │ │ ┌─────────────────────┐ │ │ │ │
+ │ │ │ │ DMEM Mapper (ID 4) │<┼────+─────┘ │
+ │ │ │ │ (FalconAppifDmemmapperV3) │ │
+ │ │ │ │ - signature: "DMAP" │ │ │ │
+ │ │ │ │ - version: 0x0003 │ │ │ │
+ │ │ │ │ - Size: 64 bytes │ │ │ │
+ │ │ │ │ - cmd_in_buffer_off │ │────┼────────────┐ │
+ │ │ │ │ - cmd_in_buffer_size│ │ │ │ │
+ │ │ │ │ - cmd_out_buffer_off│ │────┼────────────┼─────┐ │
+ │ │ │ │ - cmd_out_buffer_sz │ │ │ │ │ │
+ │ │ │ │ - init_cmd │ │ │ │ │ │
+ │ │ │ │ - features │ │ │ │ │ │
+ │ │ │ │ - cmd_mask0/1 │ │ │ │ │ │
+ │ │ │ └─────────────────────┘ │ │ │ │ │
+ │ │ │ │ │ │ │ │
+ │ │ │ ┌─────────────────────┐ │ │ │ │ │
+ │ │ │ │ Command Input Buffer│<┼────┼────────────┘ │ │
+ │ │ │ │ - Command data │ │ │ │ │
+ │ │ │ │ - Arguments │ │ │ │ │
+ │ │ │ └─────────────────────┘ │ │ │ │
+ │ │ │ │ │ │ │
+ │ │ │ ┌─────────────────────┐ │ │ │ │
+ │ │ │ │ Command Output │<┼────┼──────────────────┘ │
+ │ │ │ │ Buffer │ │ │ │
+ │ │ │ │ - Results │ │ │ │
+ │ │ │ │ - Status │ │ │ │
+ │ │ │ └─────────────────────┘ │ │ │
+ │ │ └─────────────────────────┘ │ │
+ │ └─────────────────────────────────┘ │
+ │ │
+ └───────────────────────────────────────────────────────────────┘
+
+.. note::
+ The FWSEC image also plays a role in memory scrubbing (ECC initialization)
and VPR
+ (Video Protected Region) initialization as well. Before the nova-core
driver is even
+ loaded, the FWSEC image is running on the GSP in heavy-secure mode. After
the devinit
+ sequence completes, it does VRAM memory scrubbing (ECC initialization). On
consumer
+ GPUs, it scrubs only part of memory and then initiates 'async scrubbing'.
Before this
+ async scrubbing completes, the unscrubbed VRAM cannot be used for
allocation (thus DRM
+ memory allocators need to wait for this scrubbing to complete).
\ No newline at end of file
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index 91cc802ed94f..22e5712ac6b0 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -28,4 +28,5 @@ vGPU manager VFIO driver and the nova-drm driver.
core/guidelines
core/vbios
+ core/fwsec
core/todo
--
2.43.0
