Public bug reported: In the set_port_status_up() the OVN driver tries waiting for the metadata to be provisioned (15 seconds) prior to sending the event to Nova indicating that everything is done (network-vif-plugged). But there could be a race condition while trying to get that information which results in a RowNotFound being raise in the waiting loop.
Once that happens, the exception is bubbled up and the OVN driver end up not sending the event to Nova and the instance will fail to deploy (it will be stuck in BUILD state until it times out). Here's a traceback from neutron server (q-svc) when it happens: Jan 28 10:59:55.066255 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [None req-17c35f6c-ba4f-49c3-9795-77b2edd352c7 None None] OVN reports status up for port: 2694a4c0-4ff0-414d-b780-b74da2c91197 Jan 28 10:59:55.068736 ubuntu-focal-airship-kna1-0022760767 neutron-server[68995]: DEBUG neutron.wsgi [-] (68995) accepted ('10.0.1.179', 42564) {{(pid=68995) server /usr/local/lib/python3.8/dist-packages/eventlet/wsgi.py:992}} Jan 28 10:59:55.110806 ubuntu-focal-airship-kna1-0022760767 neutron-server[68995]: INFO neutron.wsgi [req-177263b2-82ba-4f7e-84c9-d5b86115b3e7 req-f545cbf1-c1f0-4bc2-9d7e-d0420a9b43ee service neutron] 10.0.1.208,10.0.1.179 "GET /v2.0/floatingips?fixed_ip_address=10.1.0.12&port_id=2694a4c0-4ff0-414d-b780-b74da2c91197 HTTP/1.1" status: 200 len: 217 time: 0.0412221 Jan 28 10:59:55.134847 ubuntu-focal-airship-kna1-0022760767 neutron-server[68995]: DEBUG neutron.wsgi [-] (68995) accepted ('10.0.1.179', 42568) {{(pid=68995) server /usr/local/lib/python3.8/dist-packages/eventlet/wsgi.py:992}} Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event [None req-17c35f6c-ba4f-49c3-9795-77b2edd352c7 None None] Unexpected exception in notify_loop: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Chassis with name=282e849e-30b0-4e7c-9df2-2d0b14050df0 Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event Traceback (most recent call last): Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/event.py", line 159, in notify_loop Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event match.run(event, row, updates) Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py", line 392, in run Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event self.driver.set_port_status_up(row.name) Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 982, in set_port_status_up Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event self._wait_for_metadata_provisioned_if_needed(port_id) Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 1116, in _wait_for_metadata_provisioned_if_needed Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event n_utils.wait_until_true( Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/common/utils.py", line 703, in wait_until_true Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event while not predicate(): Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 1118, in <lambda> Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event self._sb_ovn.get_chassis_metadata_networks(chassis), Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 795, in get_chassis_metadata_networks Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event chassis = self.lookup('Chassis', chassis_name) Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 177, in lookup Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event return self._lookup(table, record) Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 224, in _lookup Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event row = idlutils.row_by_value(self, rl.table, rl.column, record) Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event raise RowNotFound(table=table, col=column, match=match) Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Chassis with name=282e849e-30b0-4e7c-9df2-2d0b14050df0 Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event ** Affects: neutron Importance: High Assignee: Lucas Alvares Gomes (lucasagomes) Status: Confirmed ** Changed in: neutron Importance: Undecided => High ** Changed in: neutron Assignee: (unassigned) => Lucas Alvares Gomes (lucasagomes) ** Changed in: neutron Status: New => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1914394 Title: [OVN] RowNotFound exception while waiting for Chassis metadata networks Status in neutron: Confirmed Bug description: In the set_port_status_up() the OVN driver tries waiting for the metadata to be provisioned (15 seconds) prior to sending the event to Nova indicating that everything is done (network-vif-plugged). But there could be a race condition while trying to get that information which results in a RowNotFound being raise in the waiting loop. Once that happens, the exception is bubbled up and the OVN driver end up not sending the event to Nova and the instance will fail to deploy (it will be stuck in BUILD state until it times out). Here's a traceback from neutron server (q-svc) when it happens: Jan 28 10:59:55.066255 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [None req-17c35f6c-ba4f-49c3-9795-77b2edd352c7 None None] OVN reports status up for port: 2694a4c0-4ff0-414d-b780-b74da2c91197 Jan 28 10:59:55.068736 ubuntu-focal-airship-kna1-0022760767 neutron-server[68995]: DEBUG neutron.wsgi [-] (68995) accepted ('10.0.1.179', 42564) {{(pid=68995) server /usr/local/lib/python3.8/dist-packages/eventlet/wsgi.py:992}} Jan 28 10:59:55.110806 ubuntu-focal-airship-kna1-0022760767 neutron-server[68995]: INFO neutron.wsgi [req-177263b2-82ba-4f7e-84c9-d5b86115b3e7 req-f545cbf1-c1f0-4bc2-9d7e-d0420a9b43ee service neutron] 10.0.1.208,10.0.1.179 "GET /v2.0/floatingips?fixed_ip_address=10.1.0.12&port_id=2694a4c0-4ff0-414d-b780-b74da2c91197 HTTP/1.1" status: 200 len: 217 time: 0.0412221 Jan 28 10:59:55.134847 ubuntu-focal-airship-kna1-0022760767 neutron-server[68995]: DEBUG neutron.wsgi [-] (68995) accepted ('10.0.1.179', 42568) {{(pid=68995) server /usr/local/lib/python3.8/dist-packages/eventlet/wsgi.py:992}} Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event [None req-17c35f6c-ba4f-49c3-9795-77b2edd352c7 None None] Unexpected exception in notify_loop: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Chassis with name=282e849e-30b0-4e7c-9df2-2d0b14050df0 Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event Traceback (most recent call last): Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/event.py", line 159, in notify_loop Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event match.run(event, row, updates) Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py", line 392, in run Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event self.driver.set_port_status_up(row.name) Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 982, in set_port_status_up Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event self._wait_for_metadata_provisioned_if_needed(port_id) Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 1116, in _wait_for_metadata_provisioned_if_needed Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event n_utils.wait_until_true( Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/common/utils.py", line 703, in wait_until_true Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event while not predicate(): Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 1118, in <lambda> Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event self._sb_ovn.get_chassis_metadata_networks(chassis), Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 795, in get_chassis_metadata_networks Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event chassis = self.lookup('Chassis', chassis_name) Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 177, in lookup Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event return self._lookup(table, record) Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 224, in _lookup Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event row = idlutils.row_by_value(self, rl.table, rl.column, record) Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event raise RowNotFound(table=table, col=column, match=match) Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Chassis with name=282e849e-30b0-4e7c-9df2-2d0b14050df0 Jan 28 10:59:55.135218 ubuntu-focal-airship-kna1-0022760767 neutron-server[68996]: ERROR ovsdbapp.event To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1914394/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp