On Wed, Jun 28, 2023 at 12:10:49AM +0300, Parav Pandit wrote:
> Introduce group member legacy common configuration and legacy device
> configuration access read/write commands.
>
> Group member legacy registers access commands enable group owner driver
> software to access legacy registers on behalf of the guest virtual
> machine.
>
> Usecase:
> ========
> 1. A hypervisor/system needs to provide transitional
> virtio devices to the guest VM at scale of thousands,
> typically, one to eight devices per VM.
>
> 2. A hypervisor/system needs to provide such devices using a
> vendor agnostic driver in the hypervisor system.
>
> 3. A hypervisor system prefers to have single stack regardless of
> virtio device type (net/blk) and be future compatible with a
> single vfio stack using SR-IOV or other scalable device
> virtualization technology to map PCI devices to the guest VM.
> (as transitional or otherwise)
>
> Motivation/Background:
> =====================
> The existing virtio transitional PCI device is missing support for
> PCI SR-IOV based devices. Currently it does not work beyond
> PCI PF, or as software emulated device in reality. Currently it
> has below cited system level limitations:
>
> [a] PCIe spec citation:
> VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
>
> [b] cpu arch citiation:
> Intel 64 and IA-32 Architectures Software Developer’s Manual:
> The processor’s I/O address space is separate and distinct from
> the physical-memory address space. The I/O address space consists
> of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
>
> [c] PCIe spec citation:
> If a bridge implements an I/O address range,...I/O address range will be
> aligned to a 4 KB boundary.
>
> Overview:
> =========
> Above usecase requirements is solved by PCI PF group owner accessing
> its group member PCI VFs legacy registers using the administration
> commands of the group owner PCI PF.
>
> Two types of administration commands are added which read/write PCI VF
> registers.
>
> Software usage example:
> =======================
>
> 1. One way to use and map to the guest VM is by using vfio driver
> framework in Linux kernel.
>
> +----------------------+
> |pci_dev_id = 0x100X |
> +---------------|pci_rev_id = 0x0 |-----+
> |vfio device |BAR0 = I/O region | |
> | |Other attributes | |
> | +----------------------+ |
> | |
> + +--------------+ +-----------------+ |
> | |I/O BAR to AQ | | Other vfio | |
> | |rd/wr mapper& | | functionalities | |
> | | forwarder | | | |
> | +--------------+ +-----------------+ |
> | |
> +------+-------------------------+-----------+
> | |
> Config region |
> access Driver notifications
> | |
> +----+------------+ +----+------------+
> | +-----+ | | PCI VF device A |
> | | AQ |-------------+---->+-------------+ |
> | +-----+ | | | | legacy regs | |
> | PCI PF device | | | +-------------+ |
> +-----------------+ | +-----------------+
> |
> | +----+------------+
> | | PCI VF device N |
> +---->+-------------+ |
> | | legacy regs | |
> | +-------------+ |
> +-----------------+
>
> 2. Continue to use the virtio pci driver to bind to the
> listed device id and use it as in the host.
>
> 3. Use it in a light weight hypervisor to run bare-metal OS.
>
> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
> Signed-off-by: Parav Pandit <[email protected]>
> ---
> changelog:
> v6->v7:
> - changed administrative to administration
> - renamed admin-access.tex to admin-interface.tex
> - large rewrite ad generic admin commands instead of pci
> - added theory of operation section
> - added driver notification region query command
> v5->v6:
> - fixed previous missed abbreviation of LCC and LD
> v4->v5:
> - split from pci transport specific patch
> - split conformance to transport and generic sections
> - written the description of the command as generic with member
> and group device terminology
> - reflected many section names to remove VF
> - rename fields from register to region
> - avoided abbreviation for legacy, device and config
> ---
> admin-cmds-legacy-interface.tex | 204 ++++++++++++++++++++++++++++++++
> admin.tex | 14 ++-
> conformance.tex | 2 +
> 3 files changed, 219 insertions(+), 1 deletion(-)
> create mode 100644 admin-cmds-legacy-interface.tex
>
> diff --git a/admin-cmds-legacy-interface.tex b/admin-cmds-legacy-interface.tex
> new file mode 100644
> index 0000000..48d40ee
> --- /dev/null
> +++ b/admin-cmds-legacy-interface.tex
> @@ -0,0 +1,204 @@
> +\subsubsection{Legacy Interfaces}\label{sec:Basic Facilities of a Virtio
> Device / Device groups / Group
> +administration commands / Legacy Interface}
> +
> +When the member device cannot support certain transport specific
> +functionality of a legacy interface, it is difficult to compose a
> +transitional group member device. In such scenario, a owner device
> +can provide the legacy interface functionality. With such support,
> +the member device driver can utilize the group owner device driver legacy
> +interface functionality, thereby it can supply either a transitional device
> +or a legacy device for the guest virtual machine. For example a PCI VF
> +group member device do not support I/O BAR region functionality. For such
> +limitation, a PCI PF group owner device support accessing member device's
> legacy
> +configuration region using the administration commands. These administration
> +commands are executed by the driver of the owner device when requested by the
> +driver of the member device. For example, the member device driver
> +intercepts configuration accesses done by the legacy driver and forwards
> +such access request to the owner device driver; the owner device driver
> fulfil
> +such requests by executing administration commands on the group owner device.
> +
> +The group owner device support following administration commands for legacy
> +commong configuration and device configuration region access:
> +
> +\begin{enumerate}
> +\item Legacy Common Configuration Region Write Command
> +\item Legacy Common Configuration Region Read Command
> +\item Legacy Device Configuration Region Write Command
> +\item Legacy Device Configuration Region Read Command
> +\end{enumerate}
> +
> +Accessing the member device legacy regions using these commands has the same
> +effect as accessing it using the the legacy interface defined by the
> transport
> +of the member device.
> +
> +\paragraph{Legacy Common Configuration Region Write Command}\label{par:Basic
> Facilities of a Virtio Device / Device groups / Group
> +administration commands / Legacy Interface / Common Configuration Region
> Write Command}
> +
> +This command writes in a legacy common configuration region of a group
> member device.
> +This command follows \field{struct virtio_admin_cmd}.
> +This command uses following structure for \field{command_specific_data}:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_common_cfg_wr_data {
> + u8 offset; /* Starting byte offset of the common configuration
> region to write */
> + u8 region_data[];
> +};
> +\end{lstlisting}
> +
> +The driver sets command \field{opcode} to
> VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE.
> +The driver sets valid \field{offset} and associated \field{region_data}
> bytes to
> +write to the common configuration region.
> +
> +This command does not have any command specific result.
> +
> +\paragraph{Legacy Common Configuration Region Read Command}\label{par:Basic
> Facilities of a Virtio Device / Device groups / Group administration commands
> / Legacy Interface / Common Configuration Region Read Command}
> +
> +This command reads from a legacy common configuration region of a group
> member device.
> +This command follows \field{struct virtio_admin_cmd}.
> +This command uses following structure for \field{command_specific_data}:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_common_cfg_rd_data {
> + u8 offset; /* Starting byte offset of the common configuration region
> to read */
> +};
> +\end{lstlisting}
> +
> +The driver sets command \field{opcode} to
> VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ.
> +
> +The driver sets valid \field{offset} of the region from where to read
> +\field{region_data}.
> +
> +When command completes successfully, \field{command_specific_result}
> +uses following structure:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_common_cfg_rd_result {
> + u8 region_data[];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Legacy Device Configuration Region Write Command}\label{par:Basic
> Facilities of a Virtio Device / Device groups / Group administration commands
> / Legacy Interface / Device Configuration Region Write Command}
> +
> +This command writes in a legacy device configuration region of a group
> member device.
> +This command follows \field{struct virtio_admin_cmd}.
> +This command uses following structure for \field{command_specific_data}:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_dev_reg_wr_data {
> + u8 offset; /* Starting byte offset of the device configuration
> region to write */
> + u8 region_data[];
> +};
> +\end{lstlisting}
> +
> +The driver sets command \field{opcode} to
> VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_WRITE.
> +The driver sets valid \field{offset} and associated \field{region_data}
> bytes to
> +the device configuration region.
> +
> +This command does not have any command specific result.
> +
> +\paragraph{Legacy Device Configuration Region Write Command}\label{par:Basic
> Facilities of a Virtio Device / Device groups / Group administration commands
> / Legacy Interface / Device Configuration Region Read Command}
> +
> +This command reads from a legacy device configuration region of a group
> member device.
> +This command follows \field{struct virtio_admin_cmd}.
> +
> +This command uses following structure for \field{command_specific_data}:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_dev_cfg_rd_data {
> + u8 offset; /* Starting byte offset of the device configuration
> region to read */
> +};
> +\end{lstlisting}
> +
> +The driver sets command \field{opcode} to
> VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_READ.
> +The driver sets valid \field{offset} of the device configurationi region from
> +where to read \field{region_data}.
> +
> +When command completes successfully, \field{command_specific_result}
> +uses following structure:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_dev_reg_rd_result {
> + u8 region_data[];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Legacy Driver Notification Region Query}\label{par:Basic
> Facilities of a Virtio Device / Device groups / Group administration commands
> / Legacy Interface / Legacy Driver Notifications Region Query}
> +
> +Even though the driver notifications can be communicated through the
> +administration command, if the group owner device or group member device
> +supports such notifications using a memory-mapped operation or I/O operation,
> +they are sent to the device by accessing this notification region using
> memory
> +or I/O operation.
> +
> +A group owner device optionally support querying driver notifications region
> +using VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command.
> +
> +The driver sets command \field{opcode} to
> VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY.
> +This command does not have any command specific data.
> +
> +When command completes successfully, \field{command_specific_result}
> +uses following structure:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_notify_query_entry {
> + u8 region[8];
> +};
This confuses more than it clarifies. Do this:
struct virtio_admin_cmd_legacy_notify_query_entry {
union {
virtio_pci_notify_region region;
};
};
> +
> +struct virtio_admin_cmd_legacy_notify_query_result {
> + struct virtio_virtio_admin_cmd_legacy_notify_query_entry entries[];
> +};
> +\end{lstlisting}
> +
> +The driver should pick the suitable entry when multiple entries are supplied
> +by the device.
> +
> +Refer to the specific transport section for the definition of the
> +\field{region}.
Where? How does user know where to look? Add a link.
Or preferably I would just include that tex right here
to avoid the need to jump back and forth.
> +
> +\devicenormative{\paragraph}{Legacy Interface}{Basic Facilities of a Virtio
> Device / Device groups / Group administration commands / Legacy Interface}
> +
> +If the group owner device supports legacy region access for its group member
> +devices, the device MUST set all corresponding bits for commands
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ in
> +the command result of VIRTIO_ADMIN_CMD_LIST_QUERY in
> +\field{device_admin_cmd_opcodes}.
> +
> +The device MUST encode and decode legacy device specific registers using
> +little-endian format.
> +
> +The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands for the invalid offset
> +which is outside the legacy common configuration region's address range.
> +
> +The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE,
> +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands for the invalid offset
> +which is outside the legacy device specific region's address range.
> +
> +The device SHOULD support VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command for
> +driver notifications. If the group owner device supports driver
> +notifications region for its group member devices, the device MUST set
> +VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY in the command result of
> +VIRTIO_ADMIN_CMD_LIST_QUERY in \field{device_admin_cmd_opcodes}.
> +
> +When the driver accesses the legacy region of the member device using
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE, VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ
> +commands, device MUST function as if they are accessed by the legacy
> interface
> +defined by the transport of the member device.
> +
> +\drivernormative{\paragraph}{Legacy Interface}{Basic Facilities of a Virtio
> Device / Device groups / Group administration commands / Legacy Interface}
> +
> +The driver MUST encode and decode legacy device specific registers using
> +little-endian format.
> +
> +The driver SHOULD send VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands with a valid offset which
> +is in the legacy common configuration region address range.
> +
> +The driver SHOULD send commands VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ with a valid offset which is in the
> legacy
> +device specific configuration region address range.
> +
> diff --git a/admin.tex b/admin.tex
> index fd3b97d..0de26a9 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -117,7 +117,17 @@ \subsection{Group administration
> commands}\label{sec:Basic Facilities of a Virti
> \hline
> 0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands
> used for this group type \\
> \hline
> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
> +0x0002 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE & Write legacy common
> configuration region of a member device \\
> +\hline
> +0x0003 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ & Read legacy common
> configuration region of a member device \\
> +\hline
> +0x0004 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE & Write legacy device
> configuration region of a member device \\
> +\hline
> +0x0005 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ & Read legacy device
> configuration region of a member device \\
> +\hline
> +0x0006 & VIRTIO_ADMIN_CMD_LEGACY_DEV_NOTIFY_QUERY & Query notification
> region for a member device \\
> +\hline
> +0x0007 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
> \hline
> 0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a
> different structure) \\
> \hline
> @@ -286,6 +296,8 @@ \subsection{Group administration
> commands}\label{sec:Basic Facilities of a Virti
> supporting multiple group types, the list of supported commands
> might differ between different group types.
>
> +\input{admin-cmds-legacy-interface.tex}
> +
> \devicenormative{\subsubsection}{Group administration commands}{Basic
> Facilities of a Virtio Device / Device groups / Group administration commands}
>
> The device MUST validate \field{opcode}, \field{group_type} and
> diff --git a/conformance.tex b/conformance.tex
> index 01ccd69..dc00e84 100644
> --- a/conformance.tex
> +++ b/conformance.tex
> @@ -260,6 +260,8 @@ \section{Conformance Targets}\label{sec:Conformance /
> Conformance Targets}
> \item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues /
> Legacy Interfaces: A Note on Virtqueue Layout}
> \item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues /
> Legacy Interfaces: A Note on Virtqueue Endianness}
> \item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues /
> Message Framing / Legacy Interface: Message Framing}
> +\item Section \ref{devicenormative:Basic Facilities of a Virtio Device /
> Device groups / Group administration commands / Legacy Interface}
> +\item Section \ref{drivernormative:Basic Facilities of a Virtio Device /
> Device groups / Group administration commands / Legacy Interface}
> \item Section \ref{sec:General Initialization And Device Operation / Device
> Initialization / Legacy Interface: Device Initialization}
> \item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI
> Device Discovery / Legacy Interfaces: A Note on PCI Device Discovery}
> \item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI
> Device Layout / Legacy Interfaces: A Note on PCI Device Layout}
> --
> 2.26.2
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
