Public bug reported: The functional test fails randomly as:- ft1.3: neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_maintenance.TestMaintenance.test_porttesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func return f(self, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_maintenance.py", line 306, in test_port neutron_net = self._create_network('network1') File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_maintenance.py", line 68, in _create_network return self.deserialize(self.fmt, res)['network'] KeyError: 'network'
neutron server log when it fails:- 2023-10-06 08:16:33.966 37713 DEBUG neutron.db.ovn_revision_numbers_db [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] create_initial_revision uuid=6ab7bd53-277b-4133-ac32-52b1d0c90f78, type=security_groups, rev=-1 create_initial_revision /home/zuul/src/opendev.org/openstack/neutron/neutron/db/ovn_revision_numbers_db.py:108 2023-10-06 08:16:33.973 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:33.974 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run txn.results.put(txn.do_commit()) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit raise RuntimeError(msg) RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._create_security_group-11049860 security_group, after_create: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/callbacks/manager.py", line 181, in _notify_loop 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, payload=payload) 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 409, in _create_security_group 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager self._ovn_client.create_security_group(context, 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2328, in create_security_group 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager with self._nb_idl.transaction(check_error=True) as txn: 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager next(self.gen) 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t: 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager next(self.gen) 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager with self.create_transaction( 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__ 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager self.result = self.commit() 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager raise result.ex 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager txn.results.put(txn.do_commit()) 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager raise RuntimeError(msg) 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager ... 2023-10-06 08:16:34.047 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:34.048 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run txn.results.put(txn.do_commit()) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit raise RuntimeError(msg) RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Mechanism driver 'ovn' failed in create_network_postcommit: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers Traceback (most recent call last): 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 497, in _call_on_drivers 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context) 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 579, in create_network_postcommit 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers self._ovn_client.create_network(context.plugin_context, network) 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1927, in create_network 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers with self._nb_idl.transaction(check_error=True) as txn: 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers next(self.gen) 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t: 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers next(self.gen) 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers with self.create_transaction( 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__ 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers self.result = self.commit() 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers raise result.ex 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers txn.results.put(txn.do_commit()) 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers raise RuntimeError(msg) 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers 2023-10-06 08:16:34.052 37713 ERROR neutron.plugins.ml2.plugin [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] mechanism_manager.create_network_postcommit failed, deleting network '767ee503-3645-4762-b0d6-e673b3c14170': neutron.plugins.ml2.common.exceptions.MechanismDriverError ... 2023-10-06 08:16:34.153 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:34.154 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run txn.results.put(txn.do_commit()) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit raise RuntimeError(msg) RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it : neutron.plugins.ml2.common.exceptions.MechanismDriverError 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Mechanism driver 'ovn' failed in delete_network_postcommit: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers Traceback (most recent call last): 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1219, in _after_create_network 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers self.mechanism_manager.create_network_postcommit(mech_context) 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 547, in create_network_postcommit 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers self._call_on_drivers("create_network_postcommit", context) 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 513, in _call_on_drivers 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers raise ml2_exc.MechanismDriverError( 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers neutron.plugins.ml2.common.exceptions.MechanismDriverError 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers During handling of the above exception, another exception occurred: 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers Traceback (most recent call last): 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 497, in _call_on_drivers 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context) 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 631, in delete_network_postcommit 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers self._ovn_client.delete_network( 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1938, in delete_network 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers with self._nb_idl.transaction(check_error=True) as txn: 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers next(self.gen) 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t: 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers next(self.gen) 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers with self.create_transaction( 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__ 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers self.result = self.commit() 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers raise result.ex 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers txn.results.put(txn.do_commit()) 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers raise RuntimeError(msg) 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers 2023-10-06 08:16:34.162 37713 ERROR neutron.plugins.ml2.plugin [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] mechanism_manager.delete_network_postcommit failed: neutron.plugins.ml2.common.exceptions.MechanismDriverError 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] POST failed.: neutron.plugins.ml2.common.exceptions.MechanismDriverError 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last): 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/pecan/core.py", line 693, in __call__ 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.invoke_controller(controller, args, kwargs, state) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/pecan/core.py", line 584, in invoke_controller 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation result = controller(*args, **kwargs) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception(): 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception() as ectxt: 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception(): 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/utils.py", line 65, in wrapped 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/resource.py", line 163, in post 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return self.create(resources) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/resource.py", line 181, in create 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return {key: creator(*creator_args, **creator_kwargs)} 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 731, in inner 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 223, in wrapped 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f_with_retry(*args, **kwargs, 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception(): 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception() as ectxt: 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception(): 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1211, in create_network 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return self._after_create_network(context, result, mech_context) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1221, in _after_create_network 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception(): 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1219, in _after_create_network 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.mechanism_manager.create_network_postcommit(mech_context) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 547, in create_network_postcommit 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self._call_on_drivers("create_network_postcommit", context) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 513, in _call_on_drivers 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise ml2_exc.MechanismDriverError( 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation neutron.plugins.ml2.common.exceptions.MechanismDriverError 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation 2023-10-06 08:16:34.166 37713 DEBUG neutron.pecan_wsgi.hooks.notifier [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] No notification will be sent due to unsuccessful status code: 500 after /home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/hooks/notifier.py:78 ^ From https://9ce43a75e3387ceb8909-2b4f2fa211fea8445ec0f4a568f6056b.ssl.cf2.rackcdn.com/897438/1/gate/neutron- functional-with-uwsgi/7147770/controller/logs/dsvm-functional- logs.tar.gz Other example failures:- - https://5a1da1856f577d094ffa-09419825a71508c658bc83d968b5572f.ssl.cf2.rackcdn.com/890162/3/check/neutron-functional-with-uwsgi/4ad32c0/testr_results.html - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d3b/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/d3b6e71/testr_results.html - https://cf7800dce1b91328e4f8-616b34f6dd94893c803bfdcdd39959b5.ssl.cf5.rackcdn.com/897332/4/gate/neutron-functional-with-uwsgi/fe7d494/testr_results.html - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d3b/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/d3b6e71/testr_results.html - https://cf7800dce1b91328e4f8-616b34f6dd94893c803bfdcdd39959b5.ssl.cf5.rackcdn.com/897332/4/gate/neutron-functional-with-uwsgi/fe7d494/testr_results.html ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2039417 Title: [master][stable/2023.2][functional] test_maintenance.TestMaintenance tests fails randomly Status in neutron: New Bug description: The functional test fails randomly as:- ft1.3: neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_maintenance.TestMaintenance.test_porttesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func return f(self, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_maintenance.py", line 306, in test_port neutron_net = self._create_network('network1') File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_maintenance.py", line 68, in _create_network return self.deserialize(self.fmt, res)['network'] KeyError: 'network' neutron server log when it fails:- 2023-10-06 08:16:33.966 37713 DEBUG neutron.db.ovn_revision_numbers_db [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] create_initial_revision uuid=6ab7bd53-277b-4133-ac32-52b1d0c90f78, type=security_groups, rev=-1 create_initial_revision /home/zuul/src/opendev.org/openstack/neutron/neutron/db/ovn_revision_numbers_db.py:108 2023-10-06 08:16:33.973 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:33.974 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run txn.results.put(txn.do_commit()) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit raise RuntimeError(msg) RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._create_security_group-11049860 security_group, after_create: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/callbacks/manager.py", line 181, in _notify_loop 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, payload=payload) 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 409, in _create_security_group 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager self._ovn_client.create_security_group(context, 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2328, in create_security_group 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager with self._nb_idl.transaction(check_error=True) as txn: 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager next(self.gen) 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t: 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager next(self.gen) 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager with self.create_transaction( 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__ 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager self.result = self.commit() 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager raise result.ex 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager txn.results.put(txn.do_commit()) 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager raise RuntimeError(msg) 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager ... 2023-10-06 08:16:34.047 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:34.048 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run txn.results.put(txn.do_commit()) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit raise RuntimeError(msg) RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Mechanism driver 'ovn' failed in create_network_postcommit: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers Traceback (most recent call last): 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 497, in _call_on_drivers 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context) 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 579, in create_network_postcommit 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers self._ovn_client.create_network(context.plugin_context, network) 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1927, in create_network 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers with self._nb_idl.transaction(check_error=True) as txn: 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers next(self.gen) 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t: 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers next(self.gen) 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers with self.create_transaction( 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__ 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers self.result = self.commit() 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers raise result.ex 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers txn.results.put(txn.do_commit()) 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers raise RuntimeError(msg) 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers 2023-10-06 08:16:34.052 37713 ERROR neutron.plugins.ml2.plugin [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] mechanism_manager.create_network_postcommit failed, deleting network '767ee503-3645-4762-b0d6-e673b3c14170': neutron.plugins.ml2.common.exceptions.MechanismDriverError ... 2023-10-06 08:16:34.153 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:34.154 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run txn.results.put(txn.do_commit()) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit raise RuntimeError(msg) RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it : neutron.plugins.ml2.common.exceptions.MechanismDriverError 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Mechanism driver 'ovn' failed in delete_network_postcommit: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers Traceback (most recent call last): 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1219, in _after_create_network 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers self.mechanism_manager.create_network_postcommit(mech_context) 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 547, in create_network_postcommit 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers self._call_on_drivers("create_network_postcommit", context) 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 513, in _call_on_drivers 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers raise ml2_exc.MechanismDriverError( 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers neutron.plugins.ml2.common.exceptions.MechanismDriverError 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers During handling of the above exception, another exception occurred: 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers Traceback (most recent call last): 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 497, in _call_on_drivers 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context) 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 631, in delete_network_postcommit 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers self._ovn_client.delete_network( 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1938, in delete_network 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers with self._nb_idl.transaction(check_error=True) as txn: 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers next(self.gen) 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t: 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers next(self.gen) 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers with self.create_transaction( 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__ 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers self.result = self.commit() 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers raise result.ex 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers txn.results.put(txn.do_commit()) 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers raise RuntimeError(msg) 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers 2023-10-06 08:16:34.162 37713 ERROR neutron.plugins.ml2.plugin [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] mechanism_manager.delete_network_postcommit failed: neutron.plugins.ml2.common.exceptions.MechanismDriverError 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] POST failed.: neutron.plugins.ml2.common.exceptions.MechanismDriverError 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last): 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/pecan/core.py", line 693, in __call__ 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.invoke_controller(controller, args, kwargs, state) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/pecan/core.py", line 584, in invoke_controller 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation result = controller(*args, **kwargs) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception(): 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception() as ectxt: 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception(): 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/utils.py", line 65, in wrapped 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/resource.py", line 163, in post 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return self.create(resources) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/resource.py", line 181, in create 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return {key: creator(*creator_args, **creator_kwargs)} 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 731, in inner 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 223, in wrapped 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f_with_retry(*args, **kwargs, 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception(): 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception() as ectxt: 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception(): 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1211, in create_network 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return self._after_create_network(context, result, mech_context) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1221, in _after_create_network 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception(): 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1219, in _after_create_network 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.mechanism_manager.create_network_postcommit(mech_context) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 547, in create_network_postcommit 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self._call_on_drivers("create_network_postcommit", context) 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 513, in _call_on_drivers 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise ml2_exc.MechanismDriverError( 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation neutron.plugins.ml2.common.exceptions.MechanismDriverError 2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation 2023-10-06 08:16:34.166 37713 DEBUG neutron.pecan_wsgi.hooks.notifier [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] No notification will be sent due to unsuccessful status code: 500 after /home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/hooks/notifier.py:78 ^ From https://9ce43a75e3387ceb8909-2b4f2fa211fea8445ec0f4a568f6056b.ssl.cf2.rackcdn.com/897438/1/gate/neutron- functional-with-uwsgi/7147770/controller/logs/dsvm-functional- logs.tar.gz Other example failures:- - https://5a1da1856f577d094ffa-09419825a71508c658bc83d968b5572f.ssl.cf2.rackcdn.com/890162/3/check/neutron-functional-with-uwsgi/4ad32c0/testr_results.html - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d3b/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/d3b6e71/testr_results.html - https://cf7800dce1b91328e4f8-616b34f6dd94893c803bfdcdd39959b5.ssl.cf5.rackcdn.com/897332/4/gate/neutron-functional-with-uwsgi/fe7d494/testr_results.html - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d3b/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/d3b6e71/testr_results.html - https://cf7800dce1b91328e4f8-616b34f6dd94893c803bfdcdd39959b5.ssl.cf5.rackcdn.com/897332/4/gate/neutron-functional-with-uwsgi/fe7d494/testr_results.html To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2039417/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp