Re: [ovs-discuss] controller's role mismatch?

Fri, 22 Apr 2016 15:40:41 -0700 

Hello,

I had to use 2 switches/bridges to reproduce the problem. Logs in attachments.

Just for the time orientation:
Enabling 2 masters for 2 switches (controller firstly sent slave automatically, 
and when it finds out that it is the first connection from that device, it 
sends master then)

2016-04-22T08:48:53.341Z|00080|vconn|DBG|tcp:10.25.2.14:6653: received: 
OFPT_ROLE_REQUEST (OF1.3) (xid=0x0): role=nochange
2016-04-22T08:48:53.341Z|00081|vconn|DBG|tcp:10.25.2.14:6653: sent (Success): 
OFPT_ROLE_REPLY (OF1.3) (xid=0x0): role=equal
2016-04-22T08:48:53.343Z|00082|vconn|DBG|tcp:10.25.2.14:6653: received: 
OFPT_ROLE_REQUEST (OF1.3) (xid=0x1): role=slave generation_id=0
2016-04-22T08:48:53.343Z|00083|vconn|DBG|tcp:10.25.2.14:6653: sent (Success): 
OFPT_ROLE_REPLY (OF1.3) (xid=0x1): role=slave generation_id=0
2016-04-22T08:48:53.465Z|00095|vconn|DBG|tcp:10.25.2.13:6653: received: 
OFPT_ROLE_REQUEST (OF1.3) (xid=0x0): role=nochange
2016-04-22T08:48:53.465Z|00096|vconn|DBG|tcp:10.25.2.13:6653: sent (Success): 
OFPT_ROLE_REPLY (OF1.3) (xid=0x0): role=equal
2016-04-22T08:48:53.467Z|00097|vconn|DBG|tcp:10.25.2.13:6653: received: 
OFPT_ROLE_REQUEST (OF1.3) (xid=0x1): role=slave generation_id=0
2016-04-22T08:48:53.467Z|00098|vconn|DBG|tcp:10.25.2.13:6653: sent (Success): 
OFPT_ROLE_REPLY (OF1.3) (xid=0x1): role=slave generation_id=0
2016-04-22T08:48:53.501Z|00099|vconn|DBG|tcp:10.25.2.14:6653: received: 
OFPT_ROLE_REQUEST (OF1.3) (xid=0x2): role=nochange
2016-04-22T08:48:53.501Z|00100|vconn|DBG|tcp:10.25.2.14:6653: sent (Success): 
OFPT_ROLE_REPLY (OF1.3) (xid=0x2): role=slave generation_id=0
2016-04-22T08:48:53.503Z|00101|vconn|DBG|tcp:10.25.2.14:6653: received: 
OFPT_ROLE_REQUEST (OF1.3) (xid=0x3): role=master generation_id=1
2016-04-22T08:48:53.503Z|00102|vconn|DBG|tcp:10.25.2.14:6653: sent (Success): 
OFPT_ROLE_REPLY (OF1.3) (xid=0x3): role=master generation_id=1
2016-04-22T08:48:54.121Z|00142|vconn|DBG|tcp:10.25.2.13:6653: received: 
OFPT_ROLE_REQUEST (OF1.3) (xid=0x3): role=nochange
2016-04-22T08:48:54.121Z|00143|vconn|DBG|tcp:10.25.2.13:6653: sent (Success): 
OFPT_ROLE_REPLY (OF1.3) (xid=0x3): role=slave generation_id=0
2016-04-22T08:48:54.122Z|00144|vconn|DBG|tcp:10.25.2.13:6653: received: 
OFPT_ROLE_REQUEST (OF1.3) (xid=0x4): role=master generation_id=1
2016-04-22T08:48:54.123Z|00145|vconn|DBG|tcp:10.25.2.13:6653: sent (Success): 
OFPT_ROLE_REPLY (OF1.3) (xid=0x4): role=master generation_id=1

Enabling the rest 4 slaves  (2 slaves each switch)
2016-04-22T08:48:56.718Z|00194|vconn|DBG|tcp:10.25.2.12:6653: received: 
OFPT_ROLE_REQUEST (OF1.3) (xid=0x0): role=nochange
2016-04-22T08:48:56.718Z|00195|vconn|DBG|tcp:10.25.2.12:6653: sent (Success): 
OFPT_ROLE_REPLY (OF1.3) (xid=0x0): role=equal generation_id=1
2016-04-22T08:48:56.721Z|00196|vconn|DBG|tcp:10.25.2.12:6653: received: 
OFPT_ROLE_REQUEST (OF1.3) (xid=0x1): role=slave generation_id=2
2016-04-22T08:48:56.721Z|00197|vconn|DBG|tcp:10.25.2.12:6653: sent (Success): 
OFPT_ROLE_REPLY (OF1.3) (xid=0x1): role=slave generation_id=2
2016-04-22T08:48:56.877Z|00209|vconn|DBG|tcp:10.25.2.13:6653: received: 
OFPT_ROLE_REQUEST (OF1.3) (xid=0x0): role=nochange
2016-04-22T08:48:56.877Z|00210|vconn|DBG|tcp:10.25.2.13:6653: sent (Success): 
OFPT_ROLE_REPLY (OF1.3) (xid=0x0): role=equal generation_id=2
2016-04-22T08:48:56.879Z|00211|vconn|DBG|tcp:10.25.2.13:6653: received: 
OFPT_ROLE_REQUEST (OF1.3) (xid=0x1): role=slave generation_id=3
2016-04-22T08:48:56.879Z|00212|vconn|DBG|tcp:10.25.2.13:6653: sent (Success): 
OFPT_ROLE_REPLY (OF1.3) (xid=0x1): role=slave generation_id=3
2016-04-22T08:48:57.054Z|00224|vconn|DBG|tcp:10.25.2.12:6653: received: 
OFPT_ROLE_REQUEST (OF1.3) (xid=0x0): role=nochange
2016-04-22T08:48:57.054Z|00225|vconn|DBG|tcp:10.25.2.12:6653: sent (Success): 
OFPT_ROLE_REPLY (OF1.3) (xid=0x0): role=equal generation_id=1
2016-04-22T08:48:57.055Z|00226|vconn|DBG|tcp:10.25.2.12:6653: received: 
OFPT_ROLE_REQUEST (OF1.3) (xid=0x1): role=slave generation_id=2
2016-04-22T08:48:57.055Z|00227|vconn|DBG|tcp:10.25.2.12:6653: sent (Success): 
OFPT_ROLE_REPLY (OF1.3) (xid=0x1): role=slave generation_id=2
2016-04-22T08:48:57.207Z|00239|vconn|DBG|tcp:10.25.2.14:6653: received: 
OFPT_ROLE_REQUEST (OF1.3) (xid=0x0): role=nochange
2016-04-22T08:48:57.207Z|00240|vconn|DBG|tcp:10.25.2.14:6653: sent (Success): 
OFPT_ROLE_REPLY (OF1.3) (xid=0x0): role=equal generation_id=2
2016-04-22T08:48:57.209Z|00241|vconn|DBG|tcp:10.25.2.14:6653: received: 
OFPT_ROLE_REQUEST (OF1.3) (xid=0x1): role=slave generation_id=3
2016-04-22T08:48:57.209Z|00242|vconn|DBG|tcp:10.25.2.14:6653: sent (Success): 
OFPT_ROLE_REPLY (OF1.3) (xid=0x1): role=slave generation_id=3

Peter Gubka

-----Original Message-----
From: Ben Pfaff [mailto:b...@ovn.org] 
Sent: Thursday, April 21, 2016 6:56 PM
To: Peter Gubka -X (pgubka - PANTHEON TECHNOLOGIES at Cisco) <pgu...@cisco.com>
Cc: b...@openvswitch.org
Subject: Re: [ovs-discuss] controller's role mismatch?

On Thu, Apr 21, 2016 at 02:55:46PM +0000, Peter Gubka -X (pgubka - PANTHEON 
TECHNOLOGIES at Cisco) wrote:
> Is something wrong with my approach or is it a bug in openvswitch?
> Is role parameter reliable?

Please enable OpenFlow debug logging with "-vvconn" or "ovs-appctl vlog/set 
vconn", then reproduce the problem, then send us the ovs-vswitchd log.

1)   starting network
------------------------------

mn --topo linear,2 --switch ovsk,protocols=OpenFlow13 

*** Creating network
*** Adding controller
*** Adding hosts:
h1 h2 
*** Adding switches:
s1 s2 
*** Adding links:
(h1, s1) (h2, s2) (s2, s1) 
*** Configuring hosts
h1 h2 
*** Starting controller
c0 
*** Starting 2 switches
s1 s2 ...
*** Starting CLI:
mininet>



2) ovsdb content after setting "unable to connect" controllers
-------------------------------------------------------------
BRIDGE TABLE
_uuid               : a98c6064-32b0-479d-93b2-79b70572d47d
controller          : [39b73e18-587a-4ab9-ba47-5c93e542a3f2, 
59f37413-1717-45e3-be34-d9d371e4d661, 7bac130f-2551-4155-b7c8-1388fa5a42aa]
datapath_id         : "0000000000000001"
datapath_type       : ""
external_ids        : {}
fail_mode           : secure
flood_vlans         : []
flow_tables         : {}
ipfix               : []
mirrors             : []
name                : "s1"
netflow             : []
other_config        : {datapath-id="0000000000000001", disable-in-band="true"}
ports               : [19fc60ac-efea-4b98-8402-d43581eb2b49, 
70f599a6-4d05-4fce-bbd0-1ca8b4e3f63e, b84a2cc7-ab32-4ac1-b2bc-fc6f951ca3bf]
protocols           : ["OpenFlow13"]
sflow               : []
status              : {}
stp_enable          : false

_uuid               : 31041433-80d1-4b87-b989-bb1114ae4b87
controller          : [068bde79-61dc-4d58-a0de-7e48747f9d5a, 
2a461df2-f132-4e3b-a474-a2d69288223f, bc36f4df-6fbf-4e98-84f4-16fbbce96cd7]
datapath_id         : "0000000000000002"
datapath_type       : ""
external_ids        : {}
fail_mode           : secure
flood_vlans         : []
flow_tables         : {}
ipfix               : []
mirrors             : []
name                : "s2"
netflow             : []
other_config        : {datapath-id="0000000000000002", disable-in-band="true"}
ports               : [56821f97-c297-46ff-8770-97e70b072df6, 
af98fe0b-d56f-4352-9851-b7250e898596, b976db80-91a3-4b74-ae9a-7dd3a010bfef]
protocols           : ["OpenFlow13"]
sflow               : []
status              : {}
stp_enable          : false



CONTROLLER TABLE

_uuid               : 39b73e18-587a-4ab9-ba47-5c93e542a3f2
connection_mode     : []
controller_burst_limit: []
controller_rate_limit: []
enable_async_messages: []
external_ids        : {}
inactivity_probe    : []
is_connected        : false
local_gateway       : []
local_ip            : []
local_netmask       : []
max_backoff         : []
other_config        : {}
role                : []
status              : {}
target              : "tcp:10.25.2.14:6654"

_uuid               : 068bde79-61dc-4d58-a0de-7e48747f9d5a
connection_mode     : []
controller_burst_limit: []
controller_rate_limit: []
enable_async_messages: []
external_ids        : {}
inactivity_probe    : []
is_connected        : false
local_gateway       : []
local_ip            : []
local_netmask       : []
max_backoff         : []
other_config        : {}
role                : []
status              : {}
target              : "tcp:10.25.2.12:6654"

_uuid               : 2a461df2-f132-4e3b-a474-a2d69288223f
connection_mode     : []
controller_burst_limit: []
controller_rate_limit: []
enable_async_messages: []
external_ids        : {}
inactivity_probe    : []
is_connected        : false
local_gateway       : []
local_ip            : []
local_netmask       : []
max_backoff         : []
other_config        : {}
role                : []
status              : {}
target              : "tcp:10.25.2.14:6654"

_uuid               : bc36f4df-6fbf-4e98-84f4-16fbbce96cd7
connection_mode     : []
controller_burst_limit: []
controller_rate_limit: []
enable_async_messages: []
external_ids        : {}
inactivity_probe    : []
is_connected        : false
local_gateway       : []
local_ip            : []
local_netmask       : []
max_backoff         : []
other_config        : {}
role                : []
status              : {}
target              : "tcp:10.25.2.13:6654"

_uuid               : 7bac130f-2551-4155-b7c8-1388fa5a42aa
connection_mode     : []
controller_burst_limit: []
controller_rate_limit: []
enable_async_messages: []
external_ids        : {}
inactivity_probe    : []
is_connected        : false
local_gateway       : []
local_ip            : []
local_netmask       : []
max_backoff         : []
other_config        : {}
role                : []
status              : {}
target              : "tcp:10.25.2.13:6654"

_uuid               : 59f37413-1717-45e3-be34-d9d371e4d661
connection_mode     : []
controller_burst_limit: []
controller_rate_limit: []
enable_async_messages: []
external_ids        : {}
inactivity_probe    : []
is_connected        : false
local_gateway       : []
local_ip            : []
local_netmask       : []
max_backoff         : []
other_config        : {}
role                : []
status              : {}
target              : "tcp:10.25.2.12:6654"
mininet>




3)  Enabling master controllers for bot switches
----------------------------------------------------
s1 -> 10.25.2.14
ovs-vsctl set Controller 39b73e18-587a-4ab9-ba47-5c93e542a3f2 
target="tcp\:10.25.2.14\:6653"
s2-> 10.25.2.13
ovs-vsctl set Controller bc36f4df-6fbf-4e98-84f4-16fbbce96cd7 
target="tcp\:10.25.2.13\:6653"


4) db content few seconds later - we see that 2 enabled controllers are masters
------------------------------------------

BRIDGE
_uuid               : a98c6064-32b0-479d-93b2-79b70572d47d
controller          : [39b73e18-587a-4ab9-ba47-5c93e542a3f2, 
59f37413-1717-45e3-be34-d9d371e4d661, 7bac130f-2551-4155-b7c8-1388fa5a42aa]
datapath_id         : "0000000000000001"
datapath_type       : ""
external_ids        : {}
fail_mode           : secure
flood_vlans         : []
flow_tables         : {}
ipfix               : []
mirrors             : []
name                : "s1"
netflow             : []
other_config        : {datapath-id="0000000000000001", disable-in-band="true"}
ports               : [19fc60ac-efea-4b98-8402-d43581eb2b49, 
70f599a6-4d05-4fce-bbd0-1ca8b4e3f63e, b84a2cc7-ab32-4ac1-b2bc-fc6f951ca3bf]
protocols           : ["OpenFlow13"]
sflow               : []
status              : {}
stp_enable          : false

_uuid               : 31041433-80d1-4b87-b989-bb1114ae4b87
controller          : [068bde79-61dc-4d58-a0de-7e48747f9d5a, 
2a461df2-f132-4e3b-a474-a2d69288223f, bc36f4df-6fbf-4e98-84f4-16fbbce96cd7]
datapath_id         : "0000000000000002"
datapath_type       : ""
external_ids        : {}
fail_mode           : secure
flood_vlans         : []
flow_tables         : {}
ipfix               : []
mirrors             : []
name                : "s2"
netflow             : []
other_config        : {datapath-id="0000000000000002", disable-in-band="true"}
ports               : [56821f97-c297-46ff-8770-97e70b072df6, 
af98fe0b-d56f-4352-9851-b7250e898596, b976db80-91a3-4b74-ae9a-7dd3a010bfef]
protocols           : ["OpenFlow13"]
sflow               : []
status              : {}
stp_enable          : false

CONTROLLER

_uuid               : 39b73e18-587a-4ab9-ba47-5c93e542a3f2
connection_mode     : []
controller_burst_limit: []
controller_rate_limit: []
enable_async_messages: []
external_ids        : {}
inactivity_probe    : []
is_connected        : true
local_gateway       : []
local_ip            : []
local_netmask       : []
max_backoff         : []
other_config        : {}
role                : master
status              : {sec_since_connect="2", state=ACTIVE}
target              : "tcp:10.25.2.14:6653"

_uuid               : 068bde79-61dc-4d58-a0de-7e48747f9d5a
connection_mode     : []
controller_burst_limit: []
controller_rate_limit: []
enable_async_messages: []
external_ids        : {}
inactivity_probe    : []
is_connected        : false
local_gateway       : []
local_ip            : []
local_netmask       : []
max_backoff         : []
other_config        : {}
role                : other
status              : {last_error="Permission denied", state=BACKOFF}
target              : "tcp:10.25.2.12:6654"

_uuid               : 2a461df2-f132-4e3b-a474-a2d69288223f
connection_mode     : []
controller_burst_limit: []
controller_rate_limit: []
enable_async_messages: []
external_ids        : {}
inactivity_probe    : []
is_connected        : false
local_gateway       : []
local_ip            : []
local_netmask       : []
max_backoff         : []
other_config        : {}
role                : other
status              : {last_error="Permission denied", state=BACKOFF}
target              : "tcp:10.25.2.14:6654"

_uuid               : bc36f4df-6fbf-4e98-84f4-16fbbce96cd7
connection_mode     : []
controller_burst_limit: []
controller_rate_limit: []
enable_async_messages: []
external_ids        : {}
inactivity_probe    : []
is_connected        : true
local_gateway       : []
local_ip            : []
local_netmask       : []
max_backoff         : []
other_config        : {}
role                : master
status              : {sec_since_connect="2", state=ACTIVE}
target              : "tcp:10.25.2.13:6653"

_uuid               : 7bac130f-2551-4155-b7c8-1388fa5a42aa
connection_mode     : []
controller_burst_limit: []
controller_rate_limit: []
enable_async_messages: []
external_ids        : {}
inactivity_probe    : []
is_connected        : false
local_gateway       : []
local_ip            : []
local_netmask       : []
max_backoff         : []
other_config        : {}
role                : other
status              : {last_error="Permission denied", state=BACKOFF}
target              : "tcp:10.25.2.13:6654"

_uuid               : 59f37413-1717-45e3-be34-d9d371e4d661
connection_mode     : []
controller_burst_limit: []
controller_rate_limit: []
enable_async_messages: []
external_ids        : {}
inactivity_probe    : []
is_connected        : false
local_gateway       : []
local_ip            : []
local_netmask       : []
max_backoff         : []
other_config        : {}
role                : other
status              : {last_error="Permission denied", state=BACKOFF}
target              : "tcp:10.25.2.12:6654"


5)  enabling other controllers for both switches
-------------------------------------------------------
ovs-vsctl set Controller 59f37413-1717-45e3-be34-d9d371e4d661 
target="tcp\:10.25.2.12\:6653"
ovs-vsctl set Controller 7bac130f-2551-4155-b7c8-1388fa5a42aa 
target="tcp\:10.25.2.13\:6653"
ovs-vsctl set Controller 068bde79-61dc-4d58-a0de-7e48747f9d5a 
target="tcp\:10.25.2.12\:6653"
ovs-vsctl set Controller 2a461df2-f132-4e3b-a474-a2d69288223f 
target="tcp\:10.25.2.14\:6653"


6) DB content few seconds later
--------------------------------------------

BRIDGE

_uuid               : a98c6064-32b0-479d-93b2-79b70572d47d
controller          : [39b73e18-587a-4ab9-ba47-5c93e542a3f2, 
59f37413-1717-45e3-be34-d9d371e4d661, 7bac130f-2551-4155-b7c8-1388fa5a42aa]
datapath_id         : "0000000000000001"
datapath_type       : ""
external_ids        : {}
fail_mode           : secure
flood_vlans         : []
flow_tables         : {}
ipfix               : []
mirrors             : []
name                : "s1"
netflow             : []
other_config        : {datapath-id="0000000000000001", disable-in-band="true"}
ports               : [19fc60ac-efea-4b98-8402-d43581eb2b49, 
70f599a6-4d05-4fce-bbd0-1ca8b4e3f63e, b84a2cc7-ab32-4ac1-b2bc-fc6f951ca3bf]
protocols           : ["OpenFlow13"]
sflow               : []
status              : {}
stp_enable          : false

_uuid               : 31041433-80d1-4b87-b989-bb1114ae4b87
controller          : [068bde79-61dc-4d58-a0de-7e48747f9d5a, 
2a461df2-f132-4e3b-a474-a2d69288223f, bc36f4df-6fbf-4e98-84f4-16fbbce96cd7]
datapath_id         : "0000000000000002"
datapath_type       : ""
external_ids        : {}
fail_mode           : secure
flood_vlans         : []
flow_tables         : {}
ipfix               : []
mirrors             : []
name                : "s2"
netflow             : []
other_config        : {datapath-id="0000000000000002", disable-in-band="true"}
ports               : [56821f97-c297-46ff-8770-97e70b072df6, 
af98fe0b-d56f-4352-9851-b7250e898596, b976db80-91a3-4b74-ae9a-7dd3a010bfef]
protocols           : ["OpenFlow13"]
sflow               : []
status              : {}
stp_enable          : false


CONTROLLER

_uuid               : 39b73e18-587a-4ab9-ba47-5c93e542a3f2
connection_mode     : []
controller_burst_limit: []
controller_rate_limit: []
enable_async_messages: []
external_ids        : {}
inactivity_probe    : []
is_connected        : true
local_gateway       : []
local_ip            : []
local_netmask       : []
max_backoff         : []
other_config        : {}
role                : slave
status              : {sec_since_connect="3", state=ACTIVE}
target              : "tcp:10.25.2.14:6653"

_uuid               : 068bde79-61dc-4d58-a0de-7e48747f9d5a
connection_mode     : []
controller_burst_limit: []
controller_rate_limit: []
enable_async_messages: []
external_ids        : {}
inactivity_probe    : []
is_connected        : true
local_gateway       : []
local_ip            : []
local_netmask       : []
max_backoff         : []
other_config        : {}
role                : slave
status              : {sec_since_connect="3", state=ACTIVE}
target              : "tcp:10.25.2.12:6653"

_uuid               : 2a461df2-f132-4e3b-a474-a2d69288223f
connection_mode     : []
controller_burst_limit: []
controller_rate_limit: []
enable_async_messages: []
external_ids        : {}
inactivity_probe    : []
is_connected        : true
local_gateway       : []
local_ip            : []
local_netmask       : []
max_backoff         : []
other_config        : {}
role                : slave
status              : {sec_since_connect="3", state=ACTIVE}
target              : "tcp:10.25.2.14:6653"

_uuid               : bc36f4df-6fbf-4e98-84f4-16fbbce96cd7
connection_mode     : []
controller_burst_limit: []
controller_rate_limit: []
enable_async_messages: []
external_ids        : {}
inactivity_probe    : []
is_connected        : true
local_gateway       : []
local_ip            : []
local_netmask       : []
max_backoff         : []
other_config        : {}
role                : master
status              : {sec_since_connect="7", state=ACTIVE}
target              : "tcp:10.25.2.13:6653"

_uuid               : 7bac130f-2551-4155-b7c8-1388fa5a42aa
connection_mode     : []
controller_burst_limit: []
controller_rate_limit: []
enable_async_messages: []
external_ids        : {}
inactivity_probe    : []
is_connected        : true
local_gateway       : []
local_ip            : []
local_netmask       : []
max_backoff         : []
other_config        : {}
role                : master
status              : {sec_since_connect="7", state=ACTIVE}
target              : "tcp:10.25.2.13:6653"

_uuid               : 59f37413-1717-45e3-be34-d9d371e4d661
connection_mode     : []
controller_burst_limit: []
controller_rate_limit: []
enable_async_messages: []
external_ids        : {}
inactivity_probe    : []
is_connected        : true
local_gateway       : []
local_ip            : []
local_netmask       : []
max_backoff         : []
other_config        : {}
role                : slave
status              : {sec_since_connect="3", state=ACTIVE}
target              : "tcp:10.25.2.12:6653"
mininet>

Attachment: ovs-vswitchd.log
Description: ovs-vswitchd.log

_______________________________________________
discuss mailing list
discuss@openvswitch.org
http://openvswitch.org/mailman/listinfo/discuss
  • [ovs-discuss] con... Peter Gubka -X (pgubka - PANTHEON TECHNOLOGIES at Cisco)
    • Re: [ovs-dis... Ben Pfaff
      • Re: [ovs... Peter Gubka -X (pgubka - PANTHEON TECHNOLOGIES at Cisco)
        • Re: ... Ben Pfaff
          • ... Peter Gubka -X (pgubka - PANTHEON TECHNOLOGIES at Cisco)
            • ... Ben Pfaff
              • ... Peter Gubka -X (pgubka - PANTHEON TECHNOLOGIES at Cisco)
              • ... Peter Gubka -X (pgubka - PANTHEON TECHNOLOGIES at Cisco)
                • ... Ben Pfaff
                • ... Peter Gubka -X (pgubka - PANTHEON TECHNOLOGIES at Cisco)

Reply via email to