On 1 September 2016 at 00:23, Shi Xin Ruan <steve.r...@cn.ibm.com> wrote:
> Thanks, Amitabha. > > I have 1 more question: is it possible create a l3 gateway router without > SNAT/DNAT? > So traffic from GTRP will not be SNAT, but directly forward to provider > network. > I know distribute router can do this, but I want a centralized router. At the end of the day, it is a router to which you can add static routes. SNAT/DNAT is optional. There is a unit test that does this in tests/ovn.at named "ovn -- 2 HVs, 2 LRs connected via LS, gateway router" > > > Thanks again. > > > Best Regards > Steve Ruan(阮诗新) > > Tel: (86-021)60928749, > Address: 5/F, Building 10, 399 Ke Yuan Road, Zhangjiang High-Tech Park, > Shanghai 201203, PRC > 上海浦东新区张江高科园区科苑路399号10号楼6楼 > > > > > From: Amitabha Biswas/San Jose/IBM > To: "Shi Xin Ruan" <steve.r...@cn.ibm.com> > Cc: Gurucharan Shetty <g...@ovn.org>, dev@openvswitch.org, Chandra > Sekhar Vejendla/San Jose/IBM@IBMUS > Date: 09/01/2016 02:41 PM > Subject: Re: [ovs-dev] [PATCH v3 2/5] ovn: Introduce l3 gateway > router. > ------------------------------ > > > Hi Steve, > > Here is the link to the networking-ovn patch: > https://review.openstack.org/#/c/346646/ > > Thanks > Amitabha > > > > > From: "Shi Xin Ruan" <steve.r...@cn.ibm.com> > To: Gurucharan Shetty <g...@ovn.org> > Cc: dev@openvswitch.org > Date: 08/31/2016 10:51 PM > Subject: Re: [ovs-dev] [PATCH v3 2/5] ovn: Introduce l3 gateway > router. > Sent by: "dev" <dev-boun...@openvswitch.org> > ------------------------------ > > > > Hi Guru, > > > I did not find the networking-ovn plugin for l3 gateway router. > Is it still under development? > Thanks. > > > Best Regards > Steve Ruan(阮诗新) > > Tel: (86-021)60928749, > Address: 5/F, Building 10, 399 Ke Yuan Road, Zhangjiang High-Tech Park, > Shanghai 201203, PRC > 上海浦东新区张江高科园区科苑路399号10号楼6楼 > > > > > From: Gurucharan Shetty <g...@ovn.org> > To: dev@openvswitch.org > Date: 05/20/2016 01:59 PM > Subject: [ovs-dev] [PATCH v3 2/5] ovn: Introduce l3 gateway router. > Sent by: "dev" <dev-boun...@openvswitch.org> > > > > Currently OVN has distributed switches and routers. When a packet > exits a container or a VM, the entire lifecycle of the packet > through multiple switches and routers are calculated in source > chassis itself. When the destination endpoint resides on a different > chassis, the packet is sent to the other chassis and it only goes > through the egress pipeline of that chassis once and eventually to > the real destination. > > When the packet returns back, the same thing happens. The return > packet leaves the VM/container on the chassis where it resides. > The packet goes through all the switches and routers in the logical > pipleline on that chassis and then sent to the eventual destination > over the tunnel. > > The above makes the logical pipeline very flexible and easy. But, > creates a problem for cases where you need to add stateful services > (via conntrack) on switches and routers. > > For l3 gateways, we plan to leverage DNAT and SNAT functionality > and we want to apply DNAT and SNAT rules on a router. So we ideally need > the packet to go through that router in both directions in the same > chassis. To achieve this, this commit introduces a new gateway router > which is > static and can be connected to your distributed router via a switch. > > To make minimal changes in OVN's logical pipeline, this commit > tries to make the switch port connected to a l3 gateway router look like > a container/VM endpoint for every other chassis except the chassis > on which the l3 gateway router resides. On the chassis where the > gateway router resides, the connection looks just like a patch port. > > This is achieved by the doing the following: > Introduces a new type of port_binding record called 'gateway'. > On the chassis where the gateway router resides, this port behaves just > like the port of type 'patch'. The ovn-controller on that chassis > populates the "chassis" column for this record as an indication for > other ovn-controllers of its physical location. Other ovn-controllers > treat this port as they would treat a VM/Container port on a different > chassis. > > Signed-off-by: Gurucharan Shetty <g...@ovn.org> > --- > ovn/controller/binding.c | 3 +- > ovn/controller/ovn-controller.c | 5 +- > ovn/controller/patch.c | 29 ++++++- > ovn/controller/patch.h | 3 +- > ovn/northd/ovn-northd.c | 42 +++++++-- > ovn/ovn-nb.ovsschema | 9 +- > ovn/ovn-nb.xml | 15 ++++ > ovn/ovn-sb.xml | 35 +++++++- > tests/ovn.at | 184 > ++++++++++++++++++++++++++++++++++++++++ > 9 files changed, 309 insertions(+), 16 deletions(-) > > diff --git a/ovn/controller/binding.c b/ovn/controller/binding.c > index a0d8b96..e5e55b1 100644 > --- a/ovn/controller/binding.c > +++ b/ovn/controller/binding.c > @@ -200,7 +200,8 @@ binding_run(struct controller_ctx *ctx, const struct > ovsrec_bridge *br_int, > } > sbrec_port_binding_set_chassis(binding_rec, chassis_rec); > } > - } else if (chassis_rec && binding_rec->chassis == chassis_rec) { > + } else if (chassis_rec && binding_rec->chassis == chassis_rec > + && strcmp(binding_rec->type, "gateway")) { > if (ctx->ovnsb_idl_txn) { > VLOG_INFO("Releasing lport %s from this chassis.", > binding_rec->logical_port); > diff --git a/ovn/controller/ovn-controller.c > b/ovn/controller/ovn-controller.c > index 511b184..bc4c24f 100644 > --- a/ovn/controller/ovn-controller.c > +++ b/ovn/controller/ovn-controller.c > @@ -364,8 +364,9 @@ main(int argc, char *argv[]) > &local_datapaths); > } > > - if (br_int) { > - patch_run(&ctx, br_int, &local_datapaths, > &patched_datapaths); > + if (br_int && chassis_id) { > + patch_run(&ctx, br_int, chassis_id, &local_datapaths, > + &patched_datapaths); > > struct lport_index lports; > struct mcgroup_index mcgroups; > diff --git a/ovn/controller/patch.c b/ovn/controller/patch.c > index 4808146..e8abe30 100644 > --- a/ovn/controller/patch.c > +++ b/ovn/controller/patch.c > @@ -267,12 +267,28 @@ add_patched_datapath(struct hmap *patched_datapaths, > static void > add_logical_patch_ports(struct controller_ctx *ctx, > const struct ovsrec_bridge *br_int, > + const char *local_chassis_id, > struct shash *existing_ports, > struct hmap *patched_datapaths) > { > + const struct sbrec_chassis *chassis_rec; > + chassis_rec = get_chassis(ctx->ovnsb_idl, local_chassis_id); > + if (!chassis_rec) { > + return; > + } > + > const struct sbrec_port_binding *binding; > SBREC_PORT_BINDING_FOR_EACH (binding, ctx->ovnsb_idl) { > - if (!strcmp(binding->type, "patch")) { > + bool local_port = false; > + if (!strcmp(binding->type, "gateway")) { > + const char *chassis = smap_get(&binding->options, > + "gateway-chassis"); > + if (!strcmp(local_chassis_id, chassis)) { > + local_port = true; > + } > + } > + > + if (!strcmp(binding->type, "patch") || local_port) { > const char *local = binding->logical_port; > const char *peer = smap_get(&binding->options, "peer"); > if (!peer) { > @@ -287,13 +303,19 @@ add_logical_patch_ports(struct controller_ctx *ctx, > free(dst_name); > free(src_name); > add_patched_datapath(patched_datapaths, binding); > + if (local_port) { > + if (binding->chassis != chassis_rec && > ctx->ovnsb_idl_txn) { > + sbrec_port_binding_set_chassis(binding, chassis_rec); > + } > + } > } > } > } > > void > patch_run(struct controller_ctx *ctx, const struct ovsrec_bridge *br_int, > - struct hmap *local_datapaths, struct hmap *patched_datapaths) > + const char *chassis_id, struct hmap *local_datapaths, > + struct hmap *patched_datapaths) > { > if (!ctx->ovs_idl_txn) { > return; > @@ -313,7 +335,8 @@ patch_run(struct controller_ctx *ctx, const struct > ovsrec_bridge *br_int, > * 'existing_ports' any patch ports that do exist in the database and > * should be there. */ > add_bridge_mappings(ctx, br_int, &existing_ports, local_datapaths); > - add_logical_patch_ports(ctx, br_int, &existing_ports, > patched_datapaths); > + add_logical_patch_ports(ctx, br_int, chassis_id, &existing_ports, > + patched_datapaths); > > /* Now 'existing_ports' only still contains patch ports that exist in > the > * database but shouldn't. Delete them from the database. */ > diff --git a/ovn/controller/patch.h b/ovn/controller/patch.h > index d5d842e..7920a48 100644 > --- a/ovn/controller/patch.h > +++ b/ovn/controller/patch.h > @@ -27,6 +27,7 @@ struct hmap; > struct ovsrec_bridge; > > void patch_run(struct controller_ctx *, const struct ovsrec_bridge > *br_int, > - struct hmap *local_datapaths, struct hmap > *patched_datapaths); > + const char *chassis_id, struct hmap *local_datapaths, > + struct hmap *patched_datapaths); > > #endif /* ovn/patch.h */ > diff --git a/ovn/northd/ovn-northd.c b/ovn/northd/ovn-northd.c > index f469e89..7852d83 100644 > --- a/ovn/northd/ovn-northd.c > +++ b/ovn/northd/ovn-northd.c > @@ -690,11 +690,24 @@ ovn_port_update_sbrec(const struct ovn_port *op) > { > sbrec_port_binding_set_datapath(op->sb, op->od->sb); > if (op->nbr) { > - sbrec_port_binding_set_type(op->sb, "patch"); > + /* If the router is for l3 gateway, it resides on a chassis > + * and its port type is "gateway". */ > + const char *chassis = smap_get(&op->od->nbr->options, "chassis"); > + if (chassis) { > + sbrec_port_binding_set_type(op->sb, "gateway"); > + } else { > + sbrec_port_binding_set_type(op->sb, "patch"); > + } > > const char *peer = op->peer ? op->peer->key : "<error>"; > - const struct smap ids = SMAP_CONST1(&ids, "peer", peer); > - sbrec_port_binding_set_options(op->sb, &ids); > + struct smap new; > + smap_init(&new); > + smap_add(&new, "peer", peer); > + if (chassis) { > + smap_add(&new, "gateway-chassis", chassis); > + } > + sbrec_port_binding_set_options(op->sb, &new); > + smap_destroy(&new); > > sbrec_port_binding_set_parent_port(op->sb, NULL); > sbrec_port_binding_set_tag(op->sb, NULL, 0); > @@ -704,15 +717,32 @@ ovn_port_update_sbrec(const struct ovn_port *op) > sbrec_port_binding_set_type(op->sb, op->nbs->type); > sbrec_port_binding_set_options(op->sb, &op->nbs->options); > } else { > - sbrec_port_binding_set_type(op->sb, "patch"); > + const char *chassis = NULL; > + if (op->peer && op->peer->od && op->peer->od->nbr) { > + chassis = smap_get(&op->peer->od->nbr->options, > "chassis"); > + } > + > + /* A switch port connected to a gateway router is also of > + * type "gateway". */ > + if (chassis) { > + sbrec_port_binding_set_type(op->sb, "gateway"); > + } else { > + sbrec_port_binding_set_type(op->sb, "patch"); > + } > > const char *router_port = smap_get(&op->nbs->options, > "router-port"); > if (!router_port) { > router_port = "<error>"; > } > - const struct smap ids = SMAP_CONST1(&ids, "peer", > router_port); > - sbrec_port_binding_set_options(op->sb, &ids); > + struct smap new; > + smap_init(&new); > + smap_add(&new, "peer", router_port); > + if (chassis) { > + smap_add(&new, "gateway-chassis", chassis); > + } > + sbrec_port_binding_set_options(op->sb, &new); > + smap_destroy(&new); > } > sbrec_port_binding_set_parent_port(op->sb, op->nbs->parent_name); > sbrec_port_binding_set_tag(op->sb, op->nbs->tag, op->nbs->n_tag); > diff --git a/ovn/ovn-nb.ovsschema b/ovn/ovn-nb.ovsschema > index 8163f6a..fa21b30 100644 > --- a/ovn/ovn-nb.ovsschema > +++ b/ovn/ovn-nb.ovsschema > @@ -1,7 +1,7 @@ > { > "name": "OVN_Northbound", > - "version": "2.1.1", > - "cksum": "2615511875 5108", > + "version": "2.1.2", > + "cksum": "429668869 5325", > "tables": { > "Logical_Switch": { > "columns": { > @@ -78,6 +78,11 @@ > "max": "unlimited"}}, > "default_gw": {"type": {"key": "string", "min": 0, "max": > 1}}, > "enabled": {"type": {"key": "boolean", "min": 0, "max": > 1}}, > + "options": { > + "type": {"key": "string", > + "value": "string", > + "min": 0, > + "max": "unlimited"}}, > "external_ids": { > "type": {"key": "string", "value": "string", > "min": 0, "max": "unlimited"}}}, > diff --git a/ovn/ovn-nb.xml b/ovn/ovn-nb.xml > index d7fd595..d239499 100644 > --- a/ovn/ovn-nb.xml > +++ b/ovn/ovn-nb.xml > @@ -630,6 +630,21 @@ > column is set to <code>false</code>, the router is disabled. A > disabled > router has all ingress and egress traffic dropped. > </column> > + > + <group title="Options"> > + <p> > + Additional options for the logical router. > + </p> > + > + <column name="options" key="chassis"> > + If set, indicates that the logical router in question is > + non-distributed and resides in the set chassis. The same > + value is also used by <code>ovn-controller</code> to > + uniquely identify the chassis in the OVN deployment and > + comes from <code>external_ids:system-id</code> in the > + <code>Open_vSwitch</code> table of Open_vSwitch database. > + </column> > + </group> > > <group title="Common Columns"> > <column name="external_ids"> > diff --git a/ovn/ovn-sb.xml b/ovn/ovn-sb.xml > index efd2f9a..741228c 100644 > --- a/ovn/ovn-sb.xml > +++ b/ovn/ovn-sb.xml > @@ -1220,7 +1220,12 @@ tcp.flags = RST; > which <code>ovn-controller</code>/<code>ovn-controller-vtep</code> > in > turn finds out by monitoring the local hypervisor's Open_vSwitch > database, which identifies logical ports via the conventions > described > - in <code>IntegrationGuide.md</code>. > + in <code>IntegrationGuide.md</code>. (The exceptions are for > + <code>Port_Binding</code> records of <code>type</code> 'gateway', > + whose locations are identified by <code>ovn-northd</code> via > + the <code>options:gateway-chassis</code> column in this table. > + <code>ovn-controller</code> is still responsible to populate the > + <code>chassis</code> column.) > </p> > > <p> > @@ -1298,6 +1303,14 @@ tcp.flags = RST; > a logical router to a logical switch or to another logical > router. > </dd> > > + <dt><code>gateway</code></dt> > + <dd> > + One of a pair of logical ports that act as if connected by a > patch > + cable across multiple chassis. Useful for connecting a > logical > + switch with a gateway router (which is only resident on a > + particular chassis). > + </dd> > + > <dt><code>localnet</code></dt> > <dd> > A connection to a locally accessible network from each > @@ -1336,6 +1349,26 @@ tcp.flags = RST; > </column> > </group> > > + <group title="Gateway Options"> > + <p> > + These options apply to logical ports with <ref column="type"/> of > + <code>gateway</code>. > + </p> > + > + <column name="options" key="peer"> > + The <ref column="logical_port"/> in the <ref > table="Port_Binding"/> > + record for the other side of the 'gateway' port. The named <ref > + column="logical_port"/> must specify this <ref > column="logical_port"/> > + in its own <code>peer</code> option. That is, the two 'gateway' > + logical ports must have reversed <ref column="logical_port"/> and > + <code>peer</code> values. > + </column> > + > + <column name="options" key="gateway-chassis"> > + The <code>chassis</code> in which the port resides. > + </column> > + </group> > + > <group title="Localnet Options"> > <p> > These options apply to logical ports with <ref column="type"/> of > diff --git a/tests/ovn.at b/tests/ovn.at > index a827b71..9d93064 100644 > --- a/tests/ovn.at > +++ b/tests/ovn.at > @@ -2848,3 +2848,187 @@ OVS_APP_EXIT_AND_WAIT([ovs-vswitchd]) > OVS_APP_EXIT_AND_WAIT([ovsdb-server]) > > AT_CLEANUP > + > + > +AT_SETUP([ovn -- 2 HVs, 2 LRs connected via LS, gateway router]) > +AT_KEYWORDS([ovngatewayrouter]) > +AT_SKIP_IF([test $HAVE_PYTHON = no]) > +ovn_start > + > +# Logical network: > +# Two LRs - R1 and R2 that are connected to each other via LS "join" > +# in 20.0.0.0/24 network. R1 has switchess foo (192.168.1.0/24) > +# connected to it. R2 has alice (172.16.1.0/24) connected to it. > +# R2 is a gateway router. > + > + > + > +# Create two hypervisor and create OVS ports corresponding to logical > ports. > +net_add n1 > + > +sim_add hv1 > +as hv1 > +ovs-vsctl add-br br-phys > +ovn_attach n1 br-phys 192.168.0.1 > +ovs-vsctl -- add-port br-int hv1-vif1 -- \ > + set interface hv1-vif1 external-ids:iface-id=foo1 \ > + options:tx_pcap=hv1/vif1-tx.pcap \ > + options:rxq_pcap=hv1/vif1-rx.pcap \ > + ofport-request=1 > + > + > +sim_add hv2 > +as hv2 > +ovs-vsctl add-br br-phys > +ovn_attach n1 br-phys 192.168.0.2 > +ovs-vsctl -- add-port br-int hv2-vif1 -- \ > + set interface hv2-vif1 external-ids:iface-id=alice1 \ > + options:tx_pcap=hv2/vif1-tx.pcap \ > + options:rxq_pcap=hv2/vif1-rx.pcap \ > + ofport-request=1 > + > +# Pre-populate the hypervisors' ARP tables so that we don't lose any > +# packets for ARP resolution (native tunneling doesn't queue packets > +# for ARP resolution). > +ovn_populate_arp > + > +ovn-nbctl create Logical_Router name=R1 > +ovn-nbctl create Logical_Router name=R2 options:chassis="hv2" > + > +ovn-nbctl lswitch-add foo > +ovn-nbctl lswitch-add alice > +ovn-nbctl lswitch-add join > + > +# Connect foo to R1 > +ovn-nbctl -- --id=@lrp create Logical_Router_port name=foo \ > +network=192.168.1.1/24 mac=\"00:00:01:01:02:03\" -- add Logical_Router > R1 > \ > +ports @lrp -- lport-add foo rp-foo > + > +ovn-nbctl set Logical_port rp-foo type=router options:router-port=foo \ > +addresses=\"00:00:01:01:02:03\" > + > +# Connect alice to R2 > +ovn-nbctl -- --id=@lrp create Logical_Router_port name=alice \ > +network=172.16.1.1/24 mac=\"00:00:02:01:02:03\" -- add Logical_Router R2 > \ > +ports @lrp -- lport-add alice rp-alice > + > +ovn-nbctl set Logical_port rp-alice type=router options:router-port=alice > \ > +addresses=\"00:00:02:01:02:03\" > + > + > +# Connect R1 to join > +ovn-nbctl -- --id=@lrp create Logical_Router_port name=R1_join \ > +network=20.0.0.1/24 mac=\"00:00:04:01:02:03\" -- add Logical_Router R1 \ > +ports @lrp -- lport-add join r1-join > + > +ovn-nbctl set Logical_port r1-join type=router > options:router-port=R1_join \ > +addresses='"00:00:04:01:02:03"' > + > +# Connect R2 to join > +ovn-nbctl -- --id=@lrp create Logical_Router_port name=R2_join \ > +network=20.0.0.2/24 mac=\"00:00:04:01:02:04\" -- add Logical_Router R2 \ > +ports @lrp -- lport-add join r2-join > + > +ovn-nbctl set Logical_port r2-join type=router > options:router-port=R2_join \ > +addresses='"00:00:04:01:02:04"' > + > + > +#install static routes > +ovn-nbctl -- --id=@lrt create Logical_Router_Static_Route \ > +ip_prefix=172.16.1.0/24 nexthop=20.0.0.2 -- add Logical_Router \ > +R1 static_routes @lrt > + > +ovn-nbctl -- --id=@lrt create Logical_Router_Static_Route \ > +ip_prefix=192.168.1.0/24 nexthop=20.0.0.1 -- add Logical_Router \ > +R2 static_routes @lrt > + > +# Create logical port foo1 in foo > +ovn-nbctl lport-add foo foo1 \ > +-- lport-set-addresses foo1 "f0:00:00:01:02:03 192.168.1.2" > + > +# Create logical port alice1 in alice > +ovn-nbctl lport-add alice alice1 \ > +-- lport-set-addresses alice1 "f0:00:00:01:02:04 172.16.1.2" > + > + > +# Allow some time for ovn-northd and ovn-controller to catch up. > +# XXX This should be more systematic. > +sleep 2 > + > +ip_to_hex() { > + printf "%02x%02x%02x%02x" "$@" > +} > +trim_zeros() { > + sed 's/\(00\)\{1,\}$//' > +} > + > +# Send ip packets between foo1 and alice1 > +src_mac="f00000010203" > +dst_mac="000001010203" > +src_ip=`ip_to_hex 192 168 1 2` > +dst_ip=`ip_to_hex 172 16 1 2` > +packet=${dst_mac}${src_mac}08004500001c0000000040110000${src_ip}${dst_ip} > 0035111100080000 > + > +echo "---------NB dump-----" > +ovn-nbctl show > +echo "---------------------" > +ovn-nbctl list logical_router > +echo "---------------------" > +ovn-nbctl list logical_router_port > +echo "---------------------" > + > +echo "---------SB dump-----" > +ovn-sbctl list datapath_binding > +echo "---------------------" > +ovn-sbctl list port_binding > +echo "---------------------" > +ovn-sbctl dump-flows > +echo "---------------------" > +ovn-sbctl list chassis > +ovn-sbctl list encap > +echo "---------------------" > + > +echo "------ hv1 dump ----------" > +as hv1 ovs-ofctl show br-int > +as hv1 ovs-ofctl dump-flows br-int > +echo "------ hv2 dump ----------" > +as hv2 ovs-ofctl show br-int > +as hv2 ovs-ofctl dump-flows br-int > +echo "----------------------------" > + > +# Packet to Expect at alice1 > +src_mac="000002010203" > +dst_mac="f00000010204" > +src_ip=`ip_to_hex 192 168 1 2` > +dst_ip=`ip_to_hex 172 16 1 2` > +expected=${dst_mac}${src_mac}08004500001c000000003e110200${ > src_ip}${dst_ip}0035111100080000 > + > + > +as hv1 ovs-appctl netdev-dummy/receive hv1-vif1 $packet > +as hv1 ovs-appctl ofproto/trace br-int in_port=1 $packet > + > +$PYTHON "$top_srcdir/utilities/ovs-pcap.in" hv2/vif1-tx.pcap | > trim_zeros > > received1.packets > +echo $expected | trim_zeros > expout > +AT_CHECK([cat received1.packets], [0], [expout]) > + > +for sim in hv1 hv2; do > + as $sim > + OVS_APP_EXIT_AND_WAIT([ovn-controller]) > + OVS_APP_EXIT_AND_WAIT([ovs-vswitchd]) > + OVS_APP_EXIT_AND_WAIT([ovsdb-server]) > +done > + > +as ovn-sb > +OVS_APP_EXIT_AND_WAIT([ovsdb-server]) > + > +as ovn-nb > +OVS_APP_EXIT_AND_WAIT([ovsdb-server]) > + > +as northd > +OVS_APP_EXIT_AND_WAIT([ovn-northd]) > + > +as main > +OVS_APP_EXIT_AND_WAIT([ovs-vswitchd]) > +OVS_APP_EXIT_AND_WAIT([ovsdb-server]) > + > +AT_CLEANUP > -- > 1.9.1 > > _______________________________________________ > dev mailing list > dev@openvswitch.org > http://openvswitch.org/mailman/listinfo/dev > > > > > _______________________________________________ > dev mailing list > dev@openvswitch.org > http://openvswitch.org/mailman/listinfo/dev > > > > > _______________________________________________ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
