Embarrassingly minor nitpicks follow...
On Sat, Jul 23, 2016 at 3:18 PM, Darrell Ball <dlu...@gmail.com> wrote: > Some design micro-details (e.g.) register assignments) that may change > over time were moved from the ovn-architecture.7.xml document to the > OVN-DESIGN.md document. A table is added to summarize and quantify > MFF logical metadata and register usage. The OVN-DESIGN.md file was > tested using the following markdown parsers: > > https://jbt.github.io/markdown-editor/ > http://dillinger.io/ > > The new flags register usage is also documented in the new > OVN-DESIGN.md document. > > Signed-off-by: Darrell Ball <dlu...@gmail.com> > --- > ovn/OVN-DESIGN.md | 199 > +++++++++++++++++++++++++++++++++++++++++++++ > ovn/automake.mk | 1 + > ovn/ovn-architecture.7.xml | 193 > ------------------------------------------- > 3 files changed, 200 insertions(+), 193 deletions(-) > create mode 100644 ovn/OVN-DESIGN.md > > diff --git a/ovn/OVN-DESIGN.md b/ovn/OVN-DESIGN.md > new file mode 100644 > index 0000000..2162c65 > --- /dev/null > +++ b/ovn/OVN-DESIGN.md > @@ -0,0 +1,199 @@ > +OVN register and metadata usage: > +------------------------------- > + > +logical datapath field: > + > +A field that denotes the logical datapath through which a packet is being > +processed. > +*Keep the following in sync with MFF_LOG_DATAPATH in* > +*ovn/lib/logical-fields.h.* > +OVN uses the field that OpenFlow 1.1+ simply (and confusingly) calls > +'metadata' to store the logical datapath. (This field is passed across > +tunnels as part of the tunnel key.) > + > + > +logical input port field: > + > +A field that denotes the logical port from which the packet entered the > +logical datapath. > +*Keep the following in sync with MFF_LOG_INPORT in* > +*ovn/lib/logical-fields.h.* > +OVN stores this in Nicira extension register number 14. > + > +Geneve and STT tunnels pass this field as part of the tunnel key. > +Although VXLAN tunnels do not explicitly carry a logical input port, > +OVN only uses VXLAN to communicate with gateways that from OVN's > +perspective consist of only a single logical port, so that OVN can set > +the logical input port field to this one on ingress to the OVN logical > +pipeline. > + > + > +logical output port field: > + > +A field that denotes the logical port from which the packet will leave > +the logical datapath. This is initialized to 0 at the beginning of the > +logical ingress pipeline. > +*Keep the following in sync with MFF_LOG_OUTPORT in* > +*ovn/lib/logical-fields.h.* > +OVN stores this in Nicira extension register number 15. > + > +Geneve and STT tunnels pass this field as part of the tunnel key. VXLAN > +tunnels do not transmit the logical output port field. > + > + > +conntrack zone field for logical ports: > + > +A field that denotes the connection tracking zone for logical ports. > +The value only has local significance and is not meaningful between > +chassis. This is initialized to 0 at the beginning of the logical > +ingress pipeline. > +*Keep the following in sync with MFF_LOG_CT_ZONE in* > +*ovn/lib/logical-fields.h.* > +OVN stores this in Nicira extension register number 13. > + > + > +conntrack zone fields for Gateway router: > + > +Fields that denote the connection tracking zones for Gateway routers. > +These values only have local significance (only on chassis that have > +Gateway routers instantiated) and is not meaningful between > +chassis. > +*Keep the following in sync with MFF_LOG_DNAT_ZONE and* > +*MFF_LOG_DNAT_ZONE inovn/lib/logical-fields.h.* > Missing space between "in" and "ovn/lib/logical-fields.h.*" > +OVN stores the zone information for DNATting in Nicira > +extension register number 11 and zone information for SNATing in Nicira > +extension register number 12. > + > + > +flags field: > + > +Scratchpad flags that denote the pipeline state between tables. The > +values only have local significance and are not meaningful between > +chassis. This is initialized to 0 at the beginning of the logical > +ingress pipeline. > +*Keep the following in sync with MFF_LOG_FLAGS in* > +*ovn/lib/logical-fields.h.* > +OVN stores this in Nicira extension register number 10. > + > + > +VLAN ID: > + > +The VLAN ID is used as an interface between OVN and containers nested > +inside a VM (see Life Cycle of a container interface inside a VM, in > +ovn-architecture.7.xml, for more information). > + > + > +The following table summarizes the register and metadata usage for OVN. > +*Keep the following in sync with MFF_LOG_FLAGS in* > +*ovn/lib/logical-fields.h.*: > + > +``` > + > + Register/Metadata Usage Bits (Used/Total) > + ---------------- --------- -------------- > + MFF_METADATA logical datapath 24/64 > + MFF_REG0 generic scratch 32/32 > + MFF_REG1 generic scratch 32/32 > + MFF_REG2 generic scratch 32/32 > + MFF_REG3 generic scratch 32/32 > + MFF_REG4 generic scratch 32/32 > + MFF_REG5 generic scratch 32/32 > + MFF_REG6 generic scratch 32/32 > + MFF_REG7 generic scratch 32/32 > + MFF_REG8 unused 0/32 > + MFF_REG9 unused 0/32 > + MFF_REG10 flags 1/32 > + MFF_REG11 conntrack dnat zone gateway 16/32 > + MFF_REG12 conntrack snat zone gateway 16/32 > + MFF_REG13 conntrack zone logical ports 16/32 > + MFF_REG14 logical input port 15/32 > + MFF_REG15 logical output port 16/32 > + > +``` > + > +OVN Tunnel Encapsulations: > +------------------------- > + > +tunnel key: > + > +When OVN encapsulates a packet in Geneve or another tunnel, it attaches > +extra data to it to allow the receiving OVN instance to process it > +correctly. This takes different forms depending on the particular > +encapsulation, but in each case we refer to it here as the 'tunnel key'. > + > +OVN supports three types of IP tunnel encapsulations used in the IP mesh > +connecting HVs, physical switches, appliances and/or servers. OVN passes > +packet context information across tunnels. > + > +OVN annotates logical network packets that it sends from one hypervisor to > +another with the following three pieces of metadata, which are encoded in > an > +encapsulation-specific fashion. > + > + > +24-bit logical datapath identifier: > +This is derived from the tunnel_key column in the OVN Southbound > +Datapath_Binding table. > + > +15-bit logical ingress port identifier: > +ID 0 is reserved for internal use within OVN. IDs 1 through 32767, > inclusive, > +may be assigned to logical ports (see the tunnel_key column in the OVN > +Southbound Port_Binding table). > + > +16-bit logical egress port identifier: > +IDs 0 through 32767 have the same meaning as for logical ingress ports. > IDs > +32768 through 65535, inclusive, may be assigned to logical multicast > groups > +(see the tunnel_key column in the OVN Southbound Multicast_Group table). > + > +For hypervisor-to-hypervisor traffic, OVN supports only Geneve and STT > +encapsulations, for the following reasons: > + > +Only STT and Geneve support the large amounts of metadata (over 32 bits > per > +packet) that OVN uses (as described above). > + > +STT and Geneve use randomized UDP or TCP source ports that allows > efficient > ... source ports that _allow_ efficient To be fair/clear, this 'issue' was actually in the original location. ;) > +distribution among multiple paths in environments that use ECMP in their > +underlay. > + > +NICs are available to offload STT and Geneve encapsulation and > decapsulation. > + > +Due to its flexibility, the preferred encapsulation between hypervisors is > +Geneve. For Geneve encapsulation, OVN transmits the logical datapath > +identifier in the Geneve VNI. > + > +*Keep the following in sync with ovn/controller/physical.h.* > +OVN transmits the logical ingress and logical egress ports in a TLV with > +class 0x0102, type 0, and a 32-bit value encoded as follows, from MSB to > +LSB: > + > +``` > + > + 1 15 16 > + ----------------------------------------- > + | | | | > + | resv | ingress port | egress port | > + | | | | > + ----------------------------------------- > + ``` > + > +Environments whose NICs lack Geneve offload may prefer STT encapsulation > +for performance reasons. For STT encapsulation, OVN encodes all three > +pieces of logical metadata in the STT 64-bit tunnel ID as follows, from > MSB > +to LSB: > + > +``` > + > + 9 15 16 24 > + ------------------------------------------------------ > + | | | | | > + | resv | ingress port | egress port | datapath | > + | | | | | > + ------------------------------------------------------ > +``` > + > +For connecting to gateways, in addition to Geneve and STT, OVN supports > +VXLAN, because only VXLAN support is common on top-of-rack (ToR) switches. > +Currently, gateways have a feature set that matches the capabilities as > +defined by the VTEP schema, so fewer bits of metadata are necessary. In > +the future, gateways that do not support encapsulations with large amounts > +of metadata may continue to have a reduced feature set. > + > diff --git a/ovn/automake.mk b/ovn/automake.mk > index f3f40e5..39277de 100644 > --- a/ovn/automake.mk > +++ b/ovn/automake.mk > @@ -73,6 +73,7 @@ DISTCLEANFILES += ovn/ovn-architecture.7 > EXTRA_DIST += \ > ovn/TODO \ > ovn/CONTAINERS.OpenStack.md \ > + ovn/OVN-DESIGN.md \ > ovn/OVN-GW-HA.md > > # Version checking for ovn-nb.ovsschema. > diff --git a/ovn/ovn-architecture.7.xml b/ovn/ovn-architecture.7.xml > index ead4feb..e023818 100644 > --- a/ovn/ovn-architecture.7.xml > +++ b/ovn/ovn-architecture.7.xml > @@ -609,100 +609,6 @@ > </p> > > <p> > - This section mentions several data and metadata fields, for clarity > - summarized here: > - </p> > - > - <dl> > - <dt>tunnel key</dt> > - <dd> > - When OVN encapsulates a packet in Geneve or another tunnel, it > attaches > - extra data to it to allow the receiving OVN instance to process it > - correctly. This takes different forms depending on the particular > - encapsulation, but in each case we refer to it here as the ``tunnel > - key.'' See <code>Tunnel Encapsulations</code>, below, for details. > - </dd> > - > - <dt>logical datapath field</dt> > - <dd> > - A field that denotes the logical datapath through which a packet is > being > - processed. > - <!-- Keep the following in sync with MFF_LOG_DATAPATH in > - ovn/lib/logical-fields.h. --> > - OVN uses the field that OpenFlow 1.1+ simply (and confusingly) calls > - ``metadata'' to store the logical datapath. (This field is passed > across > - tunnels as part of the tunnel key.) > - </dd> > - > - <dt>logical input port field</dt> > - <dd> > - <p> > - A field that denotes the logical port from which the packet > - entered the logical datapath. > - <!-- Keep the following in sync with MFF_LOG_INPORT in > - ovn/lib/logical-fields.h. --> > - OVN stores this in Nicira extension register number 14. > - </p> > - > - <p> > - Geneve and STT tunnels pass this field as part of the tunnel key. > - Although VXLAN tunnels do not explicitly carry a logical input > port, > - OVN only uses VXLAN to communicate with gateways that from OVN's > - perspective consist of only a single logical port, so that OVN > can set > - the logical input port field to this one on ingress to the OVN > logical > - pipeline. > - </p> > - </dd> > - > - <dt>logical output port field</dt> > - <dd> > - <p> > - A field that denotes the logical port from which the packet will > - leave the logical datapath. This is initialized to 0 at the > - beginning of the logical ingress pipeline. > - <!-- Keep the following in sync with MFF_LOG_OUTPORT in > - ovn/lib/logical-fields.h. --> > - OVN stores this in Nicira extension register number 15. > - </p> > - > - <p> > - Geneve and STT tunnels pass this field as part of the tunnel key. > - VXLAN tunnels do not transmit the logical output port field. > - </p> > - </dd> > - > - <dt>conntrack zone field for logical ports</dt> > - <dd> > - A field that denotes the connection tracking zone for logical ports. > - The value only has local significance and is not meaningful between > - chassis. This is initialized to 0 at the beginning of the logical > - <!-- Keep the following in sync with MFF_LOG_CT_ZONE in > - ovn/lib/logical-fields.h. --> > - ingress pipeline. OVN stores this in Nicira extension register > - number 13. > - </dd> > - > - <dt>conntrack zone fields for Gateway router</dt> > - <dd> > - Fields that denote the connection tracking zones for Gateway > routers. > - These values only have local significance (only on chassis that have > - Gateway routers instantiated) and is not meaningful between > - chassis. OVN stores the zone information for DNATting in Nicira > - <!-- Keep the following in sync with MFF_LOG_DNAT_ZONE and > - MFF_LOG_SNAT_ZONE in ovn/lib/logical-fields.h. --> > - extension register number 11 and zone information for SNATing in > Nicira > - extension register number 12. > - </dd> > - > - <dt>VLAN ID</dt> > - <dd> > - The VLAN ID is used as an interface between OVN and containers > nested > - inside a VM (see <code>Life Cycle of a container interface inside a > - VM</code>, above, for more information). > - </dd> > - </dl> > - > - <p> > Initially, a VM or container on the ingress hypervisor sends a packet > on a > port attached to the OVN integration bridge. Then: > </p> > @@ -1003,103 +909,4 @@ > entries and the <code>Logical_Switch</code> tunnel keys. > </li> > </ol> > - > - <h1>Design Decisions</h1> > Maybe leave a bread crumb here, just to point folks to OVN-DESIGN.md ? > - > - <h2>Tunnel Encapsulations</h2> > - > - <p> > - OVN annotates logical network packets that it sends from one > hypervisor to > - another with the following three pieces of metadata, which are > encoded in > - an encapsulation-specific fashion: > - </p> > - > - <ul> > - <li> > - 24-bit logical datapath identifier, from the <code>tunnel_key</code> > - column in the OVN Southbound <code>Datapath_Binding</code> table. > - </li> > - > - <li> > - 15-bit logical ingress port identifier. ID 0 is reserved for > internal > - use within OVN. IDs 1 through 32767, inclusive, may be assigned to > - logical ports (see the <code>tunnel_key</code> column in the OVN > - Southbound <code>Port_Binding</code> table). > - </li> > - > - <li> > - 16-bit logical egress port identifier. IDs 0 through 32767 have > the same > - meaning as for logical ingress ports. IDs 32768 through 65535, > - inclusive, may be assigned to logical multicast groups (see the > - <code>tunnel_key</code> column in the OVN Southbound > - <code>Multicast_Group</code> table). > - </li> > - </ul> > - > - <p> > - For hypervisor-to-hypervisor traffic, OVN supports only Geneve and STT > - encapsulations, for the following reasons: > - </p> > - > - <ul> > - <li> > - Only STT and Geneve support the large amounts of metadata (over 32 > bits > - per packet) that OVN uses (as described above). > - </li> > - > - <li> > - STT and Geneve use randomized UDP or TCP source ports that allows > - efficient distribution among multiple paths in environments that > use ECMP > - in their underlay. > - </li> > - > - <li> > - NICs are available to offload STT and Geneve encapsulation and > - decapsulation. > - </li> > - </ul> > - > - <p> > - Due to its flexibility, the preferred encapsulation between > hypervisors is > - Geneve. For Geneve encapsulation, OVN transmits the logical datapath > - identifier in the Geneve VNI. > - > - <!-- Keep the following in sync with ovn/controller/physical.h. --> > - OVN transmits the logical ingress and logical egress ports in a TLV > with > - class 0x0102, type 0, and a 32-bit value encoded as follows, from MSB > to > - LSB: > - </p> > - > - <diagram> > - <header name=""> > - <bits name="rsv" above="1" below="0" width=".25"/> > - <bits name="ingress port" above="15" width=".75"/> > - <bits name="egress port" above="16" width=".75"/> > - </header> > - </diagram> > - > - <p> > - Environments whose NICs lack Geneve offload may prefer STT > encapsulation > - for performance reasons. For STT encapsulation, OVN encodes all three > - pieces of logical metadata in the STT 64-bit tunnel ID as follows, > from MSB > - to LSB: > - </p> > - > - <diagram> > - <header name=""> > - <bits name="reserved" above="9" below="0" width=".5"/> > - <bits name="ingress port" above="15" width=".75"/> > - <bits name="egress port" above="16" width=".75"/> > - <bits name="datapath" above="24" width="1.25"/> > - </header> > - </diagram> > - > - <p> > - For connecting to gateways, in addition to Geneve and STT, OVN > supports > - VXLAN, because only VXLAN support is common on top-of-rack (ToR) > switches. > - Currently, gateways have a feature set that matches the capabilities > as > - defined by the VTEP schema, so fewer bits of metadata are necessary. > In > - the future, gateways that do not support encapsulations with large > amounts > - of metadata may continue to have a reduced feature set. > - </p> > </manpage> > -- > 1.9.1 > > _______________________________________________ > dev mailing list > dev@openvswitch.org > http://openvswitch.org/mailman/listinfo/dev > _______________________________________________ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
