Hi,
Ok, I've made some progress on this now the patch ports are being created
and the DR port is being scheduled on the correct chassis.
But am still having issues, I'm seeing an ARP request generated by OVN
for the
OVN gateway lr_vcn2_drg being sent out untagged on the wrong br-drg port.
The NB config again is:
router f6a30345-b550-45e2-8227-b267864de24d (lr_vcn2_drg)
port lr_vcn2_drg-lsb_vcn2_drg
mac: "40:44:00:00:01:70"
networks: ["253.255.24.131/25"]
port lr_vcn2_drg-ls_vcn2_external_drg
mac: "40:44:00:00:01:80"
networks: ["254.255.0.10/28"]
gateway chassis: [sca15-rain06 sca15-rain05 sca15-rain17]
switch 42e28448-1771-49d2-b8c1-e6743c5fc3d8 (ls_vcn2_external_drg)
port ln-ls_vcn2_external_drg
type: localnet
addresses: ["unknown"]
port ls_vcn2_external_drg-lr_vcn2_drg
type: router
router-port: lr_vcn2_drg-ls_vcn2_external_drg
ovn-nbctl lr-route-list lr_vcn2_drg
IPv4 Routes
Route Table <main>:
192.17.1.0/24 253.255.24.128 dst-ip
lr_vcn2_drg-lsb_vcn2_drg
0.0.0.0/0 254.255.0.1 dst-ip
lr_vcn2_drg-ls_vcn2_external_drg
So a VM sending to a underlay IP is getting directed to the Gayway, and
on the
chassis where the DR port is schedule I see an ARP request for
254.255.0.1 but
it is not tagged :
1689092472.482785 40:44:00:00:01:80 > ff:ff:ff:ff:ff:ff, ethertype ARP
(0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has
254.255.0.1 tell 254.255.0.10, length 28
The br-drg ports are:
ovs-vsctl list-ports br-drg
bond0
patch-ln-ls_vcn2_external_drg-to-br-int
vlan3904
tcpdump reveals we can see the pkt on br-drg and on bond0 but not vlan3904.
Any ideas, how I can get the switch to send the pkt coming in on
patch-ln-ls_vcn2_external_drg-to-br-int
to vlan3904 instead of bond0 so it will get tagged?
More details on the switch config are:
ovs-vsctl list port vlan3904
_uuid : d4434fc1-dd7a-4c50-9f3d-48a2c9334979
bond_active_slave : []
bond_downdelay : 0
bond_fake_iface : false
bond_mode : []
bond_updelay : 0
cvlans : []
external_ids : {}
fake_bridge : false
interfaces : [1f409e5c-ee05-4ba6-9af6-00db87f2b041]
lacp : []
mac : []
name : vlan3904
other_config : {}
protected : false
qos : []
rstp_statistics : {}
rstp_status : {}
statistics : {}
status : {}
tag : 3904
trunks : []
vlan_mode : []
ip a sh dev vlan3904
83: vlan3904: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UNKNOWN group default qlen 1000
link/ether 12:5e:38:65:ee:8e brd ff:ff:ff:ff:ff:ff
inet 254.255.0.2/28 brd 254.255.0.15 scope global vlan3904
valid_lft forever preferred_lft forever
inet6 fe80::105e:38ff:fe65:ee8e/64 scope link
valid_lft forever preferred_lft forever
ovs-vsctl list port bond0
_uuid : 5d17098f-050d-4008-b0d0-a3bcc2f7d659
bond_active_slave : []
bond_downdelay : 0
bond_fake_iface : false
bond_mode : []
bond_updelay : 0
cvlans : []
external_ids : {}
fake_bridge : false
interfaces : [4f6caa91-ecb1-4986-9831-5b20eef23557]
lacp : []
mac : []
name : bond0
other_config : {}
protected : false
qos : []
rstp_statistics : {}
rstp_status : {}
statistics : {}
status : {}
tag : []
trunks : []
vlan_mode : []
ip a sh dev bond0
34: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc
noqueue master ovs-system state UP group default qlen 1000
link/ether 98:03:9b:2d:91:a2 brd ff:ff:ff:ff:ff:ff
inet6 fe80::9a03:9bff:fe2d:91a2/64 scope link
valid_lft forever preferred_lft forever
ovn-nbctl list Logical_Switch_Port ln-ls_vcn2_external_drg
_uuid : 0b0d620f-430d-4c09-a210-f9e2d5366630
addresses : [unknown]
dhcpv4_options : []
dhcpv6_options : []
dynamic_addresses : []
enabled : []
external_ids : {}
ha_chassis_group : []
name : ln-ls_vcn2_external_drg
options : {network_name=drgnet}
parent_name : []
port_security : []
tag : []
tag_request : []
type : localnet
up : false
ovs-vsctl list open_vswitch
_uuid : 0ccbc819-841e-410b-8fc0-b50779fa6740
bridges : [772fa763-a6c6-4725-996f-9f488ac7c19d,
94675b86-0fdf-4277-8e04-d5f0db541a53, a46624f0-4c52-4082-8ac7-05b5e726b73f]
cur_cfg : 41
datapath_types : [netdev, system]
datapaths : {system=4198daa9-b1be-41a4-af36-56de83470451}
db_version : "8.3.0"
dpdk_initialized : false
dpdk_version : none
external_ids : {hostname=sca15-rain05.us.oracle.com,
ovn-bridge=br-int, ovn-bridge-mappings="physnet:br-ext,drgnet:br-drg",
ovn-encap-ip="253.255.2.5", ovn-encap-type=geneve,
ovn-nb="tcp:253.255.0.33:6641", ovn-remote="tcp:253.255.0.33:6642",
rundir="/var/run/openvswitch", system-id=sca15-rain05}
iface_types : [bareudp, erspan, geneve, gre, gtpu, internal,
ip6erspan, ip6gre, lisp, patch, stt, system, tap, vxlan]
manager_options : []
next_cfg : 41
other_config : {vlan-limit="0"}
ovs_version : "2.16.90"
ssl : []
statistics : {}
system_type : ol
system_version : "7.7"
On 10/07/2023 12:18, Brendan Doyle via discuss wrote:
Hi Folks,
Anybody got ideas what might be wrong here or how I can debug further.
From what I see 'ln-ls_vcn2_external_drg' does have a network_name:
ovn-nbctl lsp-get-options ln-ls_vcn2_external_drg
network_name=drgnet
ovn-sbctl list Port_Binding ln-ls_vcn2_external_drg
_uuid : 5f29178d-47a3-4553-922e-773867576acf
chassis : []
datapath : 0d0069ad-053c-4a76-8d88-bd48f806a3da
encap : []
external_ids : {}
gateway_chassis : []
ha_chassis_group : []
logical_port : ln-ls_vcn2_external_drg
mac : [unknown]
nat_addresses : []
options : {network_name=drgnet}
parent_port : []
requested_chassis : []
tag : []
tunnel_key : 2
type : localnet
up : false
virtual_parent : []
And this maps to the OVS external_ids ovn-bridge-mappings on my chassis:
sca15-rain05:
ovs-vsctl get open_vswitch . external_ids
{hostname=sca15-rain05, ovn-bridge=br-int,
ovn-bridge-mappings="physnet:br-ext,drgnet:br-drg",
ovn-encap-ip="253.255.2.5", ovn-encap-type=geneve,
ovn-nb="tcp:253.255.0.33:6641", ovn-remote="tcp:253.255.0.33:6642",
rundir="/var/run/openvswitch", system-id=ca-rain05}
sca15-rain06:
ovs-vsctl get open_vswitch . external_ids
{hostname=sca15-rain06, ovn-bridge=br-int,
ovn-bridge-mappings="physnet:br-ext,drgnet:br-drg",
ovn-encap-ip="253.255.2.6", ovn-encap-type=geneve,
ovn-nb="tcp:253.255.0.33:6641", ovn-remote="tcp:253.255.0.33:6642",
rundir="/var/run/openvswitch", system-id=ca-rain06}
sca15-rain17:
ovs-vsctl get open_vswitch . external_ids
{hostname=sca15-rain17, ovn-bridge=br-int,
ovn-bridge-mappings="physnet:br-ext,drgnet:br-drg",
ovn-encap-ip="253.255.2.17", ovn-encap-type=geneve,
ovn-nb="tcp:253.255.0.33:6641", ovn-remote="tcp:253.255.0.33:6642",
rundir="/var/run/openvswitch", system-id=ca-rain17}
Thanks
Brendan
On 07/07/2023 16:08, Brendan Doyle via discuss wrote:
Hi Folks,
I'm seeing something odd when trying to use two 'br-ext' (one called
br-ext, network_name=physnet
and one called br-drg, network_name drgnet) connected to br-int, when
I try schedule a gateway
with Distributed Router port on to three chassis. My
ovn-bridge-mappings is set as follows:
ovn-bridge-mappings="physnet:br-ext,drgnet:br-drg"
The DR Port does not get scheduled I get an error in the
ovn-controller log of:
ERR|localnet port 'ln-ls_vcn2_external_drg' has no network name.
Any thoughts? There are two odd things about this
1) The error is seen not on the chassis with the highest priority,
but on the chassis with the second highest.
# ovn-nbctl list gateway_chassis
_uuid : 511b89f0-5072-4a39-8059-df93131bfe42
chassis_name : sca15-rain05
external_ids : {}
name : lr_vcn2_drg-ls_vcn2_external_drg-sca15-rain05
options : {}
priority : 40
_uuid : 97f3076a-3a99-4ea4-8b2c-4adad02cf137
chassis_name : sca15-rain17
external_ids : {}
name : lr_vcn2_drg-ls_vcn2_external_drg-sca15-rain17
options : {}
priority : 10
_uuid : be9229a3-567f-4cf1-8c94-8456f3262bfa
chassis_name : sca15-rain06
external_ids : {}
name : lr_vcn2_drg-ls_vcn2_external_drg-sca15-rain06
options : {}
priority : 15
Error is seen on sca15-rain06.
2) The localnet port does have a network_name:
#ovn-nbctl list Logical_Switch_Port ln-ls_vcn2_external_drg
_uuid : 932084ac-b1ea-485f-9041-0985bb65563b
addresses : [unknown]
dhcpv4_options : []
dhcpv6_options : []
dynamic_addresses : []
enabled : []
external_ids : {}
ha_chassis_group : []
name : ln-ls_vcn2_external_drg
options : {network_name=drgnet}
parent_name : []
port_security : []
tag : []
tag_request : []
type : localnet
up : false
My simple network is as follows:
#ovn-nbctl show
switch fe5fde2c-64b9-4ca6-8207-f00c7a01e1d6 (ls_vcn2_external_drg)
port ls_vcn2_external_drg-lr_vcn2_drg
type: router
router-port: lr_vcn2_drg-ls_vcn2_external_drg
port ln-ls_vcn2_external_drg
type: localnet
addresses: ["unknown"]
switch 1f266320-7b84-474b-8655-ff3ca4166d4c (ls_vcn2_backbone)
port lsb_vcn2_drg-lr_vcn2_drg
type: router
router-port: lr_vcn2_drg-lsb_vcn2_drg
port lsb_vcn2_net1-lr_vcn2_net1
type: router
router-port: lr_vcn2_net1-lsb_vcn2_net1
switch d759c89d-a755-4993-9984-d5b5249c93fc (ls_vcn2_net1)
port ce78fd2b-4c68-428c-baf1-71718e7f3871
addresses: ["52:54:00:d8:6e:eb 192.17.1.5"]
port 06e85cca-867a-44fc-b2c1-be62f2fb06c0
addresses: ["52:54:00:2a:7b:49 192.17.1.6"]
port ls_vcn2_net1-lr_vcn2_net1
type: router
addresses: ["40:44:00:00:00:70"]
router-port: lr_vcn2_net1-ls_vcn2_net1
router 19568142-c253-4fb1-aff9-6bd8254705ba (lr_vcn2_drg)
port lr_vcn2_drg-ls_vcn2_external_drg
mac: "40:44:00:00:01:80"
networks: ["253.255.80.6/16"]
gateway chassis: [sca15-rain05 sca15-rain17 sca15-rain06]
port lr_vcn2_drg-lsb_vcn2_drg
mac: "40:44:00:00:01:70"
networks: ["253.255.24.131/25"]
router 47940eab-92ad-4ab9-a949-00fb5714d2cb (lr_vcn2_net1)
port lr_vcn2_net1-ls_vcn2_net1
mac: "40:44:00:00:00:70"
networks: ["192.17.1.1/24"]
port lr_vcn2_net1-lsb_vcn2_net1
mac: "40:44:00:00:00:80"
networks: ["253.255.24.128/25"]
The details of the OVS switches on the chassis (not including the
third, just the one where the
DR port should have been scheduled, and the one where an attempt to
schedule it was made):
ovn-controller node sca15-rain05 (where it should have been scheduled)
===========================================================
#ovs-vsctl list-br
br-drg
br-ext
br-int
#ovs-vsctl list-ports br-int
ovn-ca-rai-0
ovn-ca-rai-1
ovn-ca-rai-2
#ovs-vsctl list-ports br-ext
bond0.3900
#ovs-vsctl list-ports br-drg
bond0
vlan3904
#ovs-vsctl list port vlan3904
_uuid : 556ffb3a-9cfd-49f8-a82b-0cf87f9bbdca
bond_active_slave : []
bond_downdelay : 0
bond_fake_iface : false
bond_mode : []
bond_updelay : 0
cvlans : []
external_ids : {}
fake_bridge : false
interfaces : [2f594c58-a456-4738-aee4-0117d3bf0239]
lacp : []
mac : []
name : vlan3904
other_config : {}
protected : false
qos : []
rstp_statistics : {}
rstp_status : {}
statistics : {}
status : {}
tag : 3904
trunks : []
vlan_mode : []
#ovs-vsctl show
dc4974ad-9c3e-4933-ba9b-6e488a2a5261
Bridge br-drg
Port vlan3904
tag: 3904
Interface vlan3904
type: internal
Port bond0
Interface bond0
Port br-drg
Interface br-drg
type: internal
Bridge br-int
fail_mode: secure
datapath_type: system
Port ovn-ca-rai-1
Interface ovn-ca-rai-1
type: geneve
options: {csum="true", key=flow,
remote_ip="253.255.0.33"}
Port ovn-ca-rai-2
Interface ovn-ca-rai-2
type: geneve
options: {csum="true", key=flow,
remote_ip="253.255.2.17"}
Port br-int
Interface br-int
type: internal
Port ovn-ca-rai-0
Interface ovn-ca-rai-0
type: geneve
options: {csum="true", key=flow,
remote_ip="253.255.2.6"}
Bridge br-ext
Port br-ext
Interface br-ext
type: internal
Port bond0.3900
Interface bond0.3900
ovs_version: "2.16.90"
#ovs-vsctl list open_vswitch
_uuid : dc4974ad-9c3e-4933-ba9b-6e488a2a5261
bridges : [0a604cbe-f0b5-4caf-93a7-f3d6a4ffdec8,
2e7b3271-a146-4584-8cc1-915be2a6e26d,
d7e3adcc-6c20-4fa9-a019-5edb09683a79]
cur_cfg : 110
datapath_types : [netdev, system]
datapaths : {system=bbb7521f-c07f-43f3-8e7a-2f0ecf6db136}
db_version : "8.3.0"
dpdk_initialized : false
dpdk_version : none
external_ids : {hostname=sca15-rain05, ovn-bridge=br-int,
ovn-bridge-mappings="physnet:br-ext,drgnet:br-drg",
ovn-encap-ip="253.255.2.5", ovn-encap-type=geneve,
ovn-nb="tcp:253.255.0.33:6641", ovn-remote="tcp:253.255.0.33:6642",
rundir="/var/run/openvswitch", system-id=ca-rain05}
iface_types : [bareudp, erspan, geneve, gre, gtpu, internal,
ip6erspan, ip6gre, lisp, patch, stt, system, tap, vxlan]
manager_options : []
next_cfg : 110
other_config : {vlan-limit="0"}
ovs_version : "2.16.90"
ssl : []
statistics : {}
system_type : ol
system_version : "7.7"
ovn-controller node sca15-rain06 (where the error was seen)
==================================================
ovn-controller.log:
ERR|localnet port 'ln-ls_vcn2_external_drg' has no network name.
#ovs-vsctl list-br
br-drg
br-ext
br-int
#ovs-vsctl list-ports br-int
ovn-ca-rai-0
ovn-ca-rai-1
ovn-ca-rai-2
#ovs-vsctl list-ports br-ext
bond0.3900
#ovs-vsctl list-ports br-drg
bond0
vlan3904
#ovs-vsctl list port vlan3904_uuid :
817bbc88-e7bd-469d-8535-2022fab63c95
bond_active_slave : []
bond_downdelay : 0
bond_fake_iface : false
bond_mode : []
bond_updelay : 0
cvlans : []
external_ids : {}
fake_bridge : false
interfaces : [e6e90eb9-61fb-44e5-aa91-41707e83708e]
lacp : []
mac : []
name : vlan3904
other_config : {}
protected : false
qos : []
rstp_statistics : {}
rstp_status : {}
statistics : {}
status : {}
tag : 3904
trunks : []
vlan_mode : []
#ovs-vsctl show
07324b42-59e5-4488-91c9-5e0655ab22c4
Bridge br-int
fail_mode: secure
datapath_type: system
Port ovn-ca-rai-0
Interface ovn-ca-rai-0
type: geneve
options: {csum="true", key=flow,
remote_ip="253.255.0.33"}
Port br-int
Interface br-int
type: internal
Port vnet1
Interface vnet1
Port vnet5
Interface vnet5
Port ovn-ca-rai-1
Interface ovn-ca-rai-1
type: geneve
options: {csum="true", key=flow,
remote_ip="253.255.2.17"}
Port vnet3
Interface vnet3
Port ovn-ca-rai-2
Interface ovn-ca-rai-2
type: geneve
options: {csum="true", key=flow,
remote_ip="253.255.2.5"}
Bridge br-ext
Port bond0.3900
Interface bond0.3900
Port br-ext
Interface br-ext
type: internal
Bridge br-drg
Port bond0
Interface bond0
Port vlan3904
tag: 3904
Interface vlan3904
type: internal
Port br-drg
Interface br-drg
type: internal
ovs_version: "2.16.90"
#ovs-vsctl list open_vswitch
_uuid : 07324b42-59e5-4488-91c9-5e0655ab22c4
bridges : [3a0bc2a9-77ef-48e1-a463-6ad705724fa8,
4900ff28-e24a-4b0b-be78-a0ab9e7cb39a,
af90a501-4a47-4c82-9404-26e3a92d5966]
cur_cfg : 59
datapath_types : [netdev, system]
datapaths : {system=2c205461-3348-4b51-9847-2d1d7a5058a9}
db_version : "8.3.0"
dpdk_initialized : false
dpdk_version : none
external_ids : {hostname=sca15-rain06, ovn-bridge=br-int,
ovn-bridge-mappings="physnet:br-ext,drgnet:br-drg",
ovn-encap-ip="253.255.2.6", ovn-encap-type=geneve,
ovn-nb="tcp:253.255.0.33:6641", ovn-remote="tcp:253.255.0.33:6642",
rundir="/var/run/openvswitch", system-id=ca-rain06}
iface_types : [bareudp, erspan, geneve, gre, gtpu, internal,
ip6erspan, ip6gre, lisp, patch, stt, system, tap, vxlan]
manager_options : []
next_cfg : 59
other_config : {vlan-limit="0"}
ovs_version : "2.16.90"
ssl : []
statistics : {}
system_type : ol
system_version : "7.7"
Thanks Brendan
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://urldefense.com/v3/__https://mail.openvswitch.org/mailman/listinfo/ovs-discuss__;!!ACWV5N9M2RV99hQ!OxCezyLdh2keudUd4RwHMrPMlm7rLokYyo2Hu9x6v3c7x6ZXQ2ud0N11b5FF1k7X1ScHf8o8-eYakoLxex5BArCMYeoVHw$
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://urldefense.com/v3/__https://mail.openvswitch.org/mailman/listinfo/ovs-discuss__;!!ACWV5N9M2RV99hQ!O6bDv-bx6q2Q2jB5hqHvR5lcLKuPHzNU9W4f7YpPvfynamHNFBi3v2d20a8IRFMNX480etbvP91FiTPMGzw6CYZAs4FIWg$
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
