diff --git a/docs/devel/vfio-user.rst b/docs/devel/vfio-user.rst index 05a788b..ea99db6 100644 --- a/docs/devel/vfio-user.rst +++ b/docs/devel/vfio-user.rst @@ -2,9 +2,9 @@ vfio-user Protocol Specification ******************************** ------------ -Version 0.1 ------------ +------------ +Version_ 0.1 +------------ .. contents:: Table of Contents @@ -207,21 +207,19 @@ protocol. Command Concurrency ------------------- -There can be multiple outstanding commands per virtual device, e.g. a frame -buffer where the guest does multiple stores to the virtual device. Commands -must be executed in the order they are received. The server can execute and -reorder non-conflicting commands in parallel, depending on the device -semantics. - -.. Note:: - For instance, a client can issue the following operations back to back - without waiting for the first two to complete: - - 1. map a DMA region - 2. trigger some device-specific operation that results in data being read - into that DMA region, and - 3. unmap the DMA region - +A client may pipeline multiple commands without waiting for previous command +replies. The server will process commands in the order they are received. +A consequence of this is if a client issues a command with the *No_reply* bit, +then subseqently issues a command without *No_reply*, the older command will +have been processed before the reply to the younger command is sent by the +server. The client must be aware of the device's capability to process concurrent +commands if pipelining is used. For example, pipelining allows multiple client +threads to concurently access device memory; the client must ensure these acceses +obey device semantics. + +An example is a frame buffer device, where the device may allow concurrent access +to different areas of video memory, but may have indeterminate behavior if concurrent +acceses are performed to command or status registers. Socket Disconnection Behavior ----------------------------- @@ -234,29 +232,14 @@ Server Disconnection A server disconnecting from the client may indicate that: 1) A virtual device has been restarted, either intentionally (e.g. because of a - device update) or unintentionally (e.g. because of a crash). In any case, - the virtual device will come back so the client should not do anything (e.g. - simply reconnect and retry failed operations). + device update) or unintentionally (e.g. because of a crash). 2) A virtual device has been shut down with no intention to be restarted. It is impossible for the client to know whether or not a failure is intermittent or innocuous and should be retried, therefore the client should -attempt to reconnect to the socket. Since an intentional server restart (e.g. -due to an upgrade) might take some time, a reasonable timeout should be used. -In cases where the disconnection is expected (e.g. the client exits), no new -commands will be sent anyway so this situation does not pose a problem. The -control stack will clean up accordingly. - -Parameterizing this behaviour by having the server advertise a reasonable -reconnect is deferred to a future version of the protocol. - -Recovering state is implementation-specific. The protocol provides all the -building blocks and does not enforce a particular mechanism: - -* Recovering state must be initiated by the client since it is the master in - the vfio-user model. Interrupts/DMA may have to be reconfigured, - unacknowledged requests may have to be re-submitted, etc. -* The server may have to persistently store device state. +reset the VFIO device when it detects the socket has been disconnected. +Error recovery will be driven by the guest's device error handling +behavior. Client Disconnection ^^^^^^^^^^^^^^^^^^^^ @@ -274,6 +257,12 @@ take no action when the client disconnects. If anything happens to the client the control stack will know about it and can clean up resources accordingly. +Live Migration +^^^^^^^^^^^^^^ +A future version of the protocol will support client live migration. This action +will require the socket to be quiesced before it is disconnected, This mechanism +will be defined when live migration support is added. + Request Retry and Response Timeout ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ A failed command is a command that has been successfully sent and has been @@ -335,48 +324,53 @@ All messages, both command messages and reply messages, are preceded by a header that contains basic information about the message. The header is followed by message-specific data described in the sections below. -+----------------+--------+-------------+ -| Name | Offset | Size | -+================+========+=============+ -| Message ID | 0 | 2 | -+----------------+--------+-------------+ -| Command | 2 | 2 | -+----------------+--------+-------------+ -| Message size | 4 | 4 | -+----------------+--------+-------------+ -| Flags | 8 | 4 | -+----------------+--------+-------------+ -| | +-----+------------+ | -| | | Bit | Definition | | -| | +=====+============+ | -| | | 0 | Reply | | -| | +-----+------------+ | -| | | 1 | No_reply | | -| | +-----+------------+ | -| | | 2 | Error | | -| | +-----+------------+ | -+----------------+--------+-------------+ -| Error | 12 | 4 | -+----------------+--------+-------------+ -| | 16 | variable | -+----------------+--------+-------------+ - -* *Message ID* identifies the message, and is used echoed in the reply message. ++----------------+---------+-------------+ +| Name | Offset | Size | ++================+=========+=============+ +| Message ID | 0 | 2 | ++----------------+---------+-------------+ +| Command | 2 | 2 | ++----------------+---------+-------------+ +| Message size | 4 | 4 | ++----------------+---------+-------------+ +| Flags | 8 | 4 | ++----------------+---------+-------------+ +| | +------+------------+ | +| | | Bit | Definition | | +| | +======+============+ | +| | | 0-3 | Type | | +| | +------+------------+ | +| | | 4 | No_reply | | +| | +------+------------+ | +| | | 5 | Error | | +| | +------+------------+ | ++----------------+---------+-------------+ +| Errno | 12 | 4 | ++----------------+---------+-------------+ +| | 16 | variable | ++----------------+---------+-------------+ + +* *Message ID* identifies the message, and is echoed in the command's reply message. * *Command* specifies the command to be executed, listed in Commands_. * *Message size* contains the size of the entire message, including the header. * *Flags* contains attributes of the message: - * The reply bit differentiates command messages from reply messages. A reply - message acknowledges a previous command with the same message ID. - * No_reply indicates that no reply is needed for this command. This is - commonly used when multiple commands are sent, and only the last needs + * The *Type* bits indicate the message type. + + * *Command* (value 0x0) indicates a command message. + * *Reply* (value 0x1) indicates a reply message acknowledging a previous + command with the same message ID. + * *No_reply* in a command message indicates that no reply is needed for this command. + This is commonly used when multiple commands are sent, and only the last needs acknowledgement. + * *Error* in a reply message indicates the command being acknowledged had + an error. In this case, the *Errno* field will be valid. -* *Error* is a UNIX errno value set only in the reply message. It is reserved in - the command message. +* *Errno* in a reply message is a UNIX errno value. It is reserved in a command message. -Each command message in Commands_ must be replied to with a reply message, -which consists of the header with the reply bit set, plus any additional data. +Each command message in Commands_ must be replied to with a reply message, unless the +message sets the *No_Reply* bit. The reply consists of the header with the *Reply* +bit set, plus any additional data. VFIO_USER_VERSION ----------------- @@ -395,7 +389,7 @@ Message format +--------------+------------------------+ | Flags | Reply bit set in reply | +--------------+------------------------+ -| Error | 0/errno | +| Errno | 0/errno | +--------------+------------------------+ | Version | JSON byte array | +--------------+------------------------+ @@ -415,6 +409,8 @@ included: | | | be empty. | +--------------+--------+---------------------------------------------------+ +.. _Version: + Versioning and Feature Support ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Upon accepting a connection, the server must send a VFIO_USER_VERSION message @@ -457,7 +453,7 @@ Message Format +--------------+------------------------+ | Flags | Reply bit set in reply | +--------------+------------------------+ -| Error | 0/errno | +| Errno | 0/errno | +--------------+------------------------+ | Table | array of table entries | +--------------+------------------------+ @@ -543,14 +539,14 @@ Message format +--------------+----------------------------+ | Flags | Reply bit set in reply | +--------------+----------------------------+ -| Error | 0/errno | +| Errno | 0/errno | +--------------+----------------------------+ | Device info | VFIO device info | +--------------+----------------------------+ This command message is sent by the client to the server to query for basic information about the device. Only the message header is needed in the command -message. The VFIO device info structure is defined in ```` +message. The VFIO device info structure is defined in ```` (``struct vfio_device_info``). VFIO device info format @@ -576,8 +572,7 @@ VFIO device info format | num_irqs | 28 | 4 | +-------------+--------+--------------------------+ -* *argz* is reserved in vfio-user, it is only used in the ioctl() VFIO - implementation. +* *argsz* is the size of the VFIO device info structure. * *flags* contains the following device attributes. * VFIO_DEVICE_FLAGS_RESET indicates that the device supports the @@ -607,14 +602,14 @@ Message format +--------------+------------------------+ | Flags | Reply bit set in reply | +--------------+------------------------+ -| Error | 0/errno | +| Errno | 0/errno | +--------------+------------------------+ | Region info | VFIO region info | +--------------+------------------------+ This command message is sent by the client to the server to query for information about device memory regions. The VFIO region info structure is -defined in ```` (``struct vfio_region_info``). Since the client +defined in ```` (``struct vfio_region_info``). Since the client does not know the size of the capabilities, the size of the reply it should expect is 48 plus any capabilities whose size is indicated in the size field of the reply header. @@ -650,8 +645,8 @@ VFIO region info format | offset | 40 | 8 | +------------+--------+------------------------------+ -* *argz* is reserved in vfio-user, it is only used in the ioctl() VFIO - implementation. +* *argsz* is the size of the VFIO region info structure plus the + size of any region capabilities returned. * *flags* are attributes of the region: * *VFIO_REGION_INFO_FLAG_READ* allows client read access to the region. @@ -668,7 +663,7 @@ VFIO region info format * *cap_offset* describes where additional region capabilities can be found. cap_offset is relative to the beginning of the VFIO region info structure. The data structure it points is a VFIO cap header defined in - ````. + ````. * *size* is the size of the region. * *offset* is the offset given to the mmap() system call for regions with the MMAP attribute. It is also used as the base offset when mapping a VFIO @@ -679,7 +674,7 @@ VFIO Region capabilities The VFIO region information can also include a capabilities list. This list is similar to a PCI capability list - each entry has a common header that identifies a capability and where the next capability in the list can be found. -The VFIO capability header format is defined in ```` (``struct +The VFIO capability header format is defined in ```` (``struct vfio_info_cap_header``). VFIO cap header format @@ -718,7 +713,7 @@ VFIO sparse mmap The only capability supported in this version of the protocol is for sparse mmap. This capability is defined when only a subrange of the region supports direct access by the client via mmap(). The VFIO sparse mmap area is defined in -```` (``struct vfio_region_sparse_mmap_area``). +```` (``struct vfio_region_sparse_mmap_area``). VFIO region info cap sparse mmap ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -742,7 +737,7 @@ VFIO region info cap sparse mmap the base offset given in the VFIO_USER_DEVICE_GET_REGION_INFO to form the offset argument of the subsequent mmap() call. -The VFIO sparse mmap area is defined in ```` (``struct +The VFIO sparse mmap area is defined in ```` (``struct vfio_region_info_cap_sparse_mmap``). VFIO_USER_DEVICE_GET_IRQ_INFO @@ -762,14 +757,14 @@ Message format +--------------+------------------------+ | Flags | Reply bit set in reply | +--------------+------------------------+ -| Error | 0/errno | +| Errno | 0/errno | +--------------+------------------------+ | IRQ info | VFIO IRQ info | +--------------+------------------------+ This command message is sent by the client to the server to query for information about device interrupt types. The VFIO IRQ info structure is -defined in ```` (``struct vfio_irq_info``). +defined in ```` (``struct vfio_irq_info``). VFIO IRQ info format ^^^^^^^^^^^^^^^^^^^^ @@ -798,8 +793,7 @@ VFIO IRQ info format | count | 28 | 4 | +-------+--------+---------------------------+ -* *argz* is reserved in vfio-user, it is only used in the ioctl() VFIO - implementation. +* *argsz* is the size of the VFIO IRQ info structure. * *flags* defines IRQ attributes: * *VFIO_IRQ_INFO_EVENTFD* indicates the IRQ type can support server eventfd @@ -834,17 +828,17 @@ Message format +--------------+------------------------+ | Flags | Reply bit set in reply | +--------------+------------------------+ -| Error | 0/errno | +| Errno | 0/errno | +--------------+------------------------+ | IRQ set | VFIO IRQ set | +--------------+------------------------+ This command message is sent by the client to the server to set actions for device interrupt types. The VFIO IRQ set structure is defined in -```` (``struct vfio_irq_set``). +```` (``struct vfio_irq_set``). -VFIO IRQ info format -^^^^^^^^^^^^^^^^^^^^ +VFIO IRQ set format +^^^^^^^^^^^^^^^^^^^ +-------+--------+------------------------------+ | Name | Offset | Size | @@ -878,8 +872,7 @@ VFIO IRQ info format | data | 36 | variable | +-------+--------+------------------------------+ -* *argz* is reserved in vfio-user, it is only used in the ioctl() VFIO - implementation. +* *argsz* is the size of the VFIO IRQ set structure, including any *data* field. * *flags* defines the action performed on the interrupt range. The DATA flags describe the data field sent in the message; the ACTION flags describe the action to be performed. The flags are mutually exclusive for both sets. @@ -909,7 +902,8 @@ VFIO IRQ info format * *start* is the start of the sub-index being set. * *count* describes the number of sub-indexes being set. As a special case, a count of 0 with data flags of VFIO_IRQ_SET_DATA_NONE disables all interrupts - of the index data is an optional field included when the + of the index. +* *data* is an optional field included when the VFIO_IRQ_SET_DATA_BOOL flag is present. It contains an array of booleans that specify whether the action is to be performed on the corresponding index. It's used when the action is only performed on a subset of the range @@ -975,8 +969,6 @@ DMA Read/Write Data * *Count* is the size of the data to be transferred. * *Data* is the data to be read or written. -Address and count can also be accessed as ``struct iovec`` from ````. - VFIO_USER_REGION_READ --------------------- @@ -994,7 +986,7 @@ Message format +--------------+------------------------+ | Flags | Reply bit set in reply | +--------------+------------------------+ -| Error | 0/errno | +| Errno | 0/errno | +--------------+------------------------+ | Read info | REGION read/write data | +--------------+------------------------+ @@ -1021,7 +1013,7 @@ Message format +--------------+------------------------+ | Flags | Reply bit set in reply | +--------------+------------------------+ -| Error | 0/errno | +| Errno | 0/errno | +--------------+------------------------+ | Write info | REGION read/write data | +--------------+------------------------+ @@ -1048,7 +1040,7 @@ Message format +--------------+------------------------+ | Flags | Reply bit set in reply | +--------------+------------------------+ -| Error | 0/errno | +| Errno | 0/errno | +--------------+------------------------+ | DMA info | DMA read/write data | +--------------+------------------------+ @@ -1075,7 +1067,7 @@ Message format +--------------+------------------------+ | Flags | Reply bit set in reply | +--------------+------------------------+ -| Error | 0/errno | +| Errno | 0/errno | +--------------+------------------------+ | DMA info | DMA read/write data | +--------------+------------------------+ @@ -1102,7 +1094,7 @@ Message format +----------------+------------------------+ | Flags | Reply bit set in reply | +----------------+------------------------+ -| Error | 0/errno | +| Errno | 0/errno | +----------------+------------------------+ | Interrupt info | | +----------------+------------------------+ @@ -1143,7 +1135,7 @@ Message format +--------------+------------------------+ | Flags | Reply bit set in reply | +--------------+------------------------+ -| Error | 0/errno | +| Errno | 0/errno | +--------------+------------------------+ This command message is sent from the client to the server to reset the device.