Some design micro-details (e.g.) register assignments) that may change over time were moved from the ovn-architecture.7.xml document to the OVN-DESIGN.md document. A table is added to summarize and quantify MFF logical metadata and register usage. The OVN-DESIGN.md file was tested using the following markdown parsers:
https://jbt.github.io/markdown-editor/ http://dillinger.io/ The new flags register usage is also documented in the new OVN-DESIGN.md document. Signed-off-by: Darrell Ball <dlu...@gmail.com> --- ovn/OVN-DESIGN.md | 185 ++++++++++++++++++++++++++++++++++++++++++++ ovn/automake.mk | 1 + ovn/ovn-architecture.7.xml | 188 --------------------------------------------- 3 files changed, 186 insertions(+), 188 deletions(-) create mode 100644 ovn/OVN-DESIGN.md diff --git a/ovn/OVN-DESIGN.md b/ovn/OVN-DESIGN.md new file mode 100644 index 0000000..ec3f0c4 --- /dev/null +++ b/ovn/OVN-DESIGN.md @@ -0,0 +1,185 @@ +OVN register and metadata usage: +------------------------------- + +logical datapath field: + +A field that denotes the logical datapath through which a packet is being +processed. +*Keep the following in sync with MFF_LOG_DATAPATH in* +*ovn/lib/logical-fields.h.* +OVN uses the field that OpenFlow 1.1+ simply (and confusingly) calls +'metadata' to store the logical datapath. (This field is passed across +tunnels as part of the tunnel key.) + + +logical input port field: + +A field that denotes the logical port from which the packet entered the +logical datapath. +*Keep the following in sync with MFF_LOG_INPORT in* +*ovn/lib/logical-fields.h.* +OVN stores this in Nicira extension register number 6. + +Geneve and STT tunnels pass this field as part of the tunnel key. +Although VXLAN tunnels do not explicitly carry a logical input port, +OVN only uses VXLAN to communicate with gateways that from OVN's +perspective consist of only a single logical port, so that OVN can set +the logical input port field to this one on ingress to the OVN logical +pipeline. + + +logical output port field: + +A field that denotes the logical port from which the packet will leave +the logical datapath. This is initialized to 0 at the beginning of the +logical ingress pipeline. +*Keep the following in sync with MFF_LOG_OUTPORT in* +*ovn/lib/logical-fields.h.* +OVN stores this in Nicira extension register number 7. + +Geneve and STT tunnels pass this field as part of the tunnel key. VXLAN +tunnels do not transmit the logical output port field. + + +conntrack zone field for logical ports: + +A field that denotes the connection tracking zone for logical ports. +The value only has local significance and is not meaningful between +chassis. This is initialized to 0 at the beginning of the logical +ingress pipeline. OVN stores this in Nicira extension register number 5. + + +conntrack zone fields for Gateway router: + +Fields that denote the connection tracking zones for Gateway routers. +These values only have local significance (only on chassis that have +Gateway routers instantiated) and is not meaningful between +chassis. OVN stores the zone information for DNATting in Nicira +extension register number 3 and zone information for SNATing in Nicira +extension register number 4. + + +flags field: + +Scratchpad flags that denote the pipeline state between tables. The +values only have local significance and are not meaningful between +chassis. This is initialized to 0 at the beginning of the logical +ingress pipeline. +*Keep the following in sync with MFF_LOG_FLAGS in* +*ovn/lib/logical-fields.h.* +OVN stores this in Nicira extension register number 2. + + +VLAN ID: + +The VLAN ID is used as an interface between OVN and containers nested +inside a VM (see Life Cycle of a container interface inside a VM, in +ovn-architecture.7.xml, for more information). + + +The following table summarizes the register and metadata usage for OVN. +*Keep the following in sync with MFF_LOG_FLAGS in* +*ovn/lib/logical-fields.h.*: + +``` + + Register/Metadata Usage Bits (Used/Total) + ---------------- --------- -------------- + MFF_METADATA logical datapath 24/64 + MFF_REG0 generic scratch 32/32 + MFF_REG1 generic scratch 32/32 + MFF_REG2 flags 1/32 + MFF_REG3 conntrack dnat zone gateway 16/32 + MFF_REG4 conntrack snat zone gateway 16/32 + MFF_REG5 conntrack zone logical ports 16/32 + MFF_REG6 logical input port 15/32 + MFF_REG7 logical output port 16/32 + +``` + +OVN Tunnel Encapsulations: +------------------------- + +tunnel key: + +When OVN encapsulates a packet in Geneve or another tunnel, it attaches +extra data to it to allow the receiving OVN instance to process it +correctly. This takes different forms depending on the particular +encapsulation, but in each case we refer to it here as the 'tunnel key'. + +OVN supports three types of IP tunnel encapsulations used in the IP mesh +connecting HVs, physical switches, appliances and/or servers. OVN passes +packet context information across tunnels. + +OVN annotates logical network packets that it sends from one hypervisor to +another with the following three pieces of metadata, which are encoded in an +encapsulation-specific fashion. + + +24-bit logical datapath identifier: +This is derived from the tunnel_key column in the OVN Southbound +Datapath_Binding table. + +15-bit logical ingress port identifier: +ID 0 is reserved for internal use within OVN. IDs 1 through 32767, inclusive, +may be assigned to logical ports (see the tunnel_key column in the OVN +Southbound Port_Binding table). + +16-bit logical egress port identifier: +IDs 0 through 32767 have the same meaning as for logical ingress ports. IDs +32768 through 65535, inclusive, may be assigned to logical multicast groups +(see the tunnel_key column in the OVN Southbound Multicast_Group table). + +For hypervisor-to-hypervisor traffic, OVN supports only Geneve and STT +encapsulations, for the following reasons: + +Only STT and Geneve support the large amounts of metadata (over 32 bits per +packet) that OVN uses (as described above). + +STT and Geneve use randomized UDP or TCP source ports that allows efficient +distribution among multiple paths in environments that use ECMP in their +underlay. + +NICs are available to offload STT and Geneve encapsulation and decapsulation. + +Due to its flexibility, the preferred encapsulation between hypervisors is +Geneve. For Geneve encapsulation, OVN transmits the logical datapath +identifier in the Geneve VNI. + +*Keep the following in sync with ovn/controller/physical.h.* +OVN transmits the logical ingress and logical egress ports in a TLV with +class 0x0102, type 0, and a 32-bit value encoded as follows, from MSB to +LSB: + +``` + + 1 15 16 + ----------------------------------------- + | | | | + | resv | ingress port | egress port | + | | | | + ----------------------------------------- + ``` + +Environments whose NICs lack Geneve offload may prefer STT encapsulation +for performance reasons. For STT encapsulation, OVN encodes all three +pieces of logical metadata in the STT 64-bit tunnel ID as follows, from MSB +to LSB: + +``` + + 9 15 16 24 + ------------------------------------------------------ + | | | | | + | resv | ingress port | egress port | datapath | + | | | | | + ------------------------------------------------------ +``` + +For connecting to gateways, in addition to Geneve and STT, OVN supports +VXLAN, because only VXLAN support is common on top-of-rack (ToR) switches. +Currently, gateways have a feature set that matches the capabilities as +defined by the VTEP schema, so fewer bits of metadata are necessary. In +the future, gateways that do not support encapsulations with large amounts +of metadata may continue to have a reduced feature set. + diff --git a/ovn/automake.mk b/ovn/automake.mk index f3f40e5..39277de 100644 --- a/ovn/automake.mk +++ b/ovn/automake.mk @@ -73,6 +73,7 @@ DISTCLEANFILES += ovn/ovn-architecture.7 EXTRA_DIST += \ ovn/TODO \ ovn/CONTAINERS.OpenStack.md \ + ovn/OVN-DESIGN.md \ ovn/OVN-GW-HA.md # Version checking for ovn-nb.ovsschema. diff --git a/ovn/ovn-architecture.7.xml b/ovn/ovn-architecture.7.xml index 72786bc..e023818 100644 --- a/ovn/ovn-architecture.7.xml +++ b/ovn/ovn-architecture.7.xml @@ -609,95 +609,6 @@ </p> <p> - This section mentions several data and metadata fields, for clarity - summarized here: - </p> - - <dl> - <dt>tunnel key</dt> - <dd> - When OVN encapsulates a packet in Geneve or another tunnel, it attaches - extra data to it to allow the receiving OVN instance to process it - correctly. This takes different forms depending on the particular - encapsulation, but in each case we refer to it here as the ``tunnel - key.'' See <code>Tunnel Encapsulations</code>, below, for details. - </dd> - - <dt>logical datapath field</dt> - <dd> - A field that denotes the logical datapath through which a packet is being - processed. - <!-- Keep the following in sync with MFF_LOG_DATAPATH in - ovn/lib/logical-fields.h. --> - OVN uses the field that OpenFlow 1.1+ simply (and confusingly) calls - ``metadata'' to store the logical datapath. (This field is passed across - tunnels as part of the tunnel key.) - </dd> - - <dt>logical input port field</dt> - <dd> - <p> - A field that denotes the logical port from which the packet - entered the logical datapath. - <!-- Keep the following in sync with MFF_LOG_INPORT in - ovn/lib/logical-fields.h. --> - OVN stores this in Nicira extension register number 6. - </p> - - <p> - Geneve and STT tunnels pass this field as part of the tunnel key. - Although VXLAN tunnels do not explicitly carry a logical input port, - OVN only uses VXLAN to communicate with gateways that from OVN's - perspective consist of only a single logical port, so that OVN can set - the logical input port field to this one on ingress to the OVN logical - pipeline. - </p> - </dd> - - <dt>logical output port field</dt> - <dd> - <p> - A field that denotes the logical port from which the packet will - leave the logical datapath. This is initialized to 0 at the - beginning of the logical ingress pipeline. - <!-- Keep the following in sync with MFF_LOG_OUTPORT in - ovn/lib/logical-fields.h. --> - OVN stores this in Nicira extension register number 7. - </p> - - <p> - Geneve and STT tunnels pass this field as part of the tunnel key. - VXLAN tunnels do not transmit the logical output port field. - </p> - </dd> - - <dt>conntrack zone field for logical ports</dt> - <dd> - A field that denotes the connection tracking zone for logical ports. - The value only has local significance and is not meaningful between - chassis. This is initialized to 0 at the beginning of the logical - ingress pipeline. OVN stores this in Nicira extension register number 5. - </dd> - - <dt>conntrack zone fields for Gateway router</dt> - <dd> - Fields that denote the connection tracking zones for Gateway routers. - These values only have local significance (only on chassis that have - Gateway routers instantiated) and is not meaningful between - chassis. OVN stores the zone information for DNATting in Nicira - extension register number 3 and zone information for SNATing in Nicira - extension register number 4. - </dd> - - <dt>VLAN ID</dt> - <dd> - The VLAN ID is used as an interface between OVN and containers nested - inside a VM (see <code>Life Cycle of a container interface inside a - VM</code>, above, for more information). - </dd> - </dl> - - <p> Initially, a VM or container on the ingress hypervisor sends a packet on a port attached to the OVN integration bridge. Then: </p> @@ -998,103 +909,4 @@ entries and the <code>Logical_Switch</code> tunnel keys. </li> </ol> - - <h1>Design Decisions</h1> - - <h2>Tunnel Encapsulations</h2> - - <p> - OVN annotates logical network packets that it sends from one hypervisor to - another with the following three pieces of metadata, which are encoded in - an encapsulation-specific fashion: - </p> - - <ul> - <li> - 24-bit logical datapath identifier, from the <code>tunnel_key</code> - column in the OVN Southbound <code>Datapath_Binding</code> table. - </li> - - <li> - 15-bit logical ingress port identifier. ID 0 is reserved for internal - use within OVN. IDs 1 through 32767, inclusive, may be assigned to - logical ports (see the <code>tunnel_key</code> column in the OVN - Southbound <code>Port_Binding</code> table). - </li> - - <li> - 16-bit logical egress port identifier. IDs 0 through 32767 have the same - meaning as for logical ingress ports. IDs 32768 through 65535, - inclusive, may be assigned to logical multicast groups (see the - <code>tunnel_key</code> column in the OVN Southbound - <code>Multicast_Group</code> table). - </li> - </ul> - - <p> - For hypervisor-to-hypervisor traffic, OVN supports only Geneve and STT - encapsulations, for the following reasons: - </p> - - <ul> - <li> - Only STT and Geneve support the large amounts of metadata (over 32 bits - per packet) that OVN uses (as described above). - </li> - - <li> - STT and Geneve use randomized UDP or TCP source ports that allows - efficient distribution among multiple paths in environments that use ECMP - in their underlay. - </li> - - <li> - NICs are available to offload STT and Geneve encapsulation and - decapsulation. - </li> - </ul> - - <p> - Due to its flexibility, the preferred encapsulation between hypervisors is - Geneve. For Geneve encapsulation, OVN transmits the logical datapath - identifier in the Geneve VNI. - - <!-- Keep the following in sync with ovn/controller/physical.h. --> - OVN transmits the logical ingress and logical egress ports in a TLV with - class 0x0102, type 0, and a 32-bit value encoded as follows, from MSB to - LSB: - </p> - - <diagram> - <header name=""> - <bits name="rsv" above="1" below="0" width=".25"/> - <bits name="ingress port" above="15" width=".75"/> - <bits name="egress port" above="16" width=".75"/> - </header> - </diagram> - - <p> - Environments whose NICs lack Geneve offload may prefer STT encapsulation - for performance reasons. For STT encapsulation, OVN encodes all three - pieces of logical metadata in the STT 64-bit tunnel ID as follows, from MSB - to LSB: - </p> - - <diagram> - <header name=""> - <bits name="reserved" above="9" below="0" width=".5"/> - <bits name="ingress port" above="15" width=".75"/> - <bits name="egress port" above="16" width=".75"/> - <bits name="datapath" above="24" width="1.25"/> - </header> - </diagram> - - <p> - For connecting to gateways, in addition to Geneve and STT, OVN supports - VXLAN, because only VXLAN support is common on top-of-rack (ToR) switches. - Currently, gateways have a feature set that matches the capabilities as - defined by the VTEP schema, so fewer bits of metadata are necessary. In - the future, gateways that do not support encapsulations with large amounts - of metadata may continue to have a reduced feature set. - </p> </manpage> -- 1.9.1 _______________________________________________ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
