Public bug reported: Test neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent.test_agent_resync_on_non_existing_bridge is failing intermitelly. Ex [0]
I just reintroduced this test [1] into the code. The failure is does not happen all the time but I can be reproduce it locally with --until- failure with multiple concurrency(big hint) and running the whole TestMetadaAgent class of tests (another hint). Like this `tox -e dsvm-functional -- neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent --until-failure --concurrency 0` When the failure happens following exception is found in the logs 2023-03-10 17:49:11.861 40848 INFO neutron.agent.ovn.metadata.agent [-] Port ovn-port-feb89eb1-fcf8-4b38-8aee-d8dd4b6b497e in datapath ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b bound to our chassis 2023-03-10 17:49:11.863 40848 INFO neutron.agent.ovn.metadata.agent [-] Provisioning metadata for network ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b 2023-03-10 17:49:11.917 40848 DEBUG neutron.agent.ovn.metadata.agent [-] Creating VETH tapovn-f376a61 in ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b namespace provision_datapath /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py:603 2023-03-10 17:49:11.923 41596 DEBUG neutron.privileged.agent.linux.ip_lib [-] Interface tapovn-f376a60 not found in namespace None get_link_id /home/zuul/src/opendev.org/openstack/neutron/neutron/privileged/agent/linux/ip_lib.py:204 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b. 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event Traceback (most recent call last): 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/event.py", line 177, in notify_loop 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event match.run(event, row, updates) 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py", line 110, in run 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event self.agent.provision_datapath(row.datapath) 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py", line 640, in provision_datapath 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event ip2.addr.add_multiple(ipv4_cidrs_to_add) 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 544, in add_multiple 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event add_ip_addresses(cidrs, self.name, self._parent.namespace, scope, 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 848, in add_ip_addresses 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event privileged.add_ip_addresses( 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event return self.channel.remote_call(name, args, kwargs, 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 215, in remote_call 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event raise exc_type(*result[2]) 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b. 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event The thing is this line of code should actually be creating the new namespace[2] so not sure why its complaining that namespace was not found. I am suspecting there is some race condition or more likely some test interferance due to test runner concurrency. [0] https://zuul.opendev.org/t/openstack/build/5d1910037db844e88cf9ef694068cf17 [1] https://review.opendev.org/c/openstack/neutron/+/875586 [2] https://github.com/openstack/neutron/blob/master/neutron/agent/ovn/metadata/agent.py#LL610-L611 ** Affects: neutron Importance: Undecided Assignee: Miro Tomaska (mtomaska) Status: New ** Changed in: neutron Assignee: (unassigned) => Miro Tomaska (mtomaska) ** Description changed: Test neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent.test_agent_resync_on_non_existing_bridge - is failing intermitelly. + is failing intermitelly. Ex [0] - I just reintroduced this test [1] into the code. The failure is does not happen all the time but I can be reproduce it locally with --until-failure with multiple concurrency(big hint) and running the whole TestMetadaAgent class of tests (another hint). Like this - - `tox -e dsvm-functional -- neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent --until-failure --concurrency 0` + I just reintroduced this test [1] into the code. The failure is does not + happen all the time but I can be reproduce it locally with --until- + failure with multiple concurrency(big hint) and running the whole + TestMetadaAgent class of tests (another hint). Like this + + `tox -e dsvm-functional -- + neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent + --until-failure --concurrency 0` When the failure happens following exception is found in the logs 2023-03-10 17:49:11.861 40848 INFO neutron.agent.ovn.metadata.agent [-] Port ovn-port-feb89eb1-fcf8-4b38-8aee-d8dd4b6b497e in datapath ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b bound to our chassis 2023-03-10 17:49:11.863 40848 INFO neutron.agent.ovn.metadata.agent [-] Provisioning metadata for network ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b 2023-03-10 17:49:11.917 40848 DEBUG neutron.agent.ovn.metadata.agent [-] Creating VETH tapovn-f376a61 in ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b namespace provision_datapath /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py:603 2023-03-10 17:49:11.923 41596 DEBUG neutron.privileged.agent.linux.ip_lib [-] Interface tapovn-f376a60 not found in namespace None get_link_id /home/zuul/src/opendev.org/openstack/neutron/neutron/privileged/agent/linux/ip_lib.py:204 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b. 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event Traceback (most recent call last): 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/event.py", line 177, in notify_loop 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event match.run(event, row, updates) 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py", line 110, in run 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event self.agent.provision_datapath(row.datapath) 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py", line 640, in provision_datapath 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event ip2.addr.add_multiple(ipv4_cidrs_to_add) 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 544, in add_multiple 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event add_ip_addresses(cidrs, self.name, self._parent.namespace, scope, 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 848, in add_ip_addresses 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event privileged.add_ip_addresses( 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event return self.channel.remote_call(name, args, kwargs, 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 215, in remote_call 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event raise exc_type(*result[2]) 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b. - 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event + 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event + The thing is this line of code should actually be creating the new + namespace[2] so not sure why its complaining that namespace was not + found. I am suspecting there is some race condition or more likely some + test interferance due to test runner concurrency. - The thing is this line of code should actually be creating the new namespace[2] so not sure why its complaining that namespace was not found. I am suspecting there is some race condition or more likely some test interferance due to test runner concurrency. - - + [0] https://zuul.opendev.org/t/openstack/build/5d1910037db844e88cf9ef694068cf17 [1] https://review.opendev.org/c/openstack/neutron/+/875586 [2] https://github.com/openstack/neutron/blob/master/neutron/agent/ovn/metadata/agent.py#LL610-L611 -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2011377 Title: test_agent_resync_on_non_existing_bridge failing intermittently sp Status in neutron: New Bug description: Test neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent.test_agent_resync_on_non_existing_bridge is failing intermitelly. Ex [0] I just reintroduced this test [1] into the code. The failure is does not happen all the time but I can be reproduce it locally with --until-failure with multiple concurrency(big hint) and running the whole TestMetadaAgent class of tests (another hint). Like this `tox -e dsvm-functional -- neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent --until-failure --concurrency 0` When the failure happens following exception is found in the logs 2023-03-10 17:49:11.861 40848 INFO neutron.agent.ovn.metadata.agent [-] Port ovn-port-feb89eb1-fcf8-4b38-8aee-d8dd4b6b497e in datapath ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b bound to our chassis 2023-03-10 17:49:11.863 40848 INFO neutron.agent.ovn.metadata.agent [-] Provisioning metadata for network ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b 2023-03-10 17:49:11.917 40848 DEBUG neutron.agent.ovn.metadata.agent [-] Creating VETH tapovn-f376a61 in ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b namespace provision_datapath /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py:603 2023-03-10 17:49:11.923 41596 DEBUG neutron.privileged.agent.linux.ip_lib [-] Interface tapovn-f376a60 not found in namespace None get_link_id /home/zuul/src/opendev.org/openstack/neutron/neutron/privileged/agent/linux/ip_lib.py:204 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b. 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event Traceback (most recent call last): 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/event.py", line 177, in notify_loop 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event match.run(event, row, updates) 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py", line 110, in run 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event self.agent.provision_datapath(row.datapath) 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py", line 640, in provision_datapath 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event ip2.addr.add_multiple(ipv4_cidrs_to_add) 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 544, in add_multiple 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event add_ip_addresses(cidrs, self.name, self._parent.namespace, scope, 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 848, in add_ip_addresses 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event privileged.add_ip_addresses( 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event return self.channel.remote_call(name, args, kwargs, 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 215, in remote_call 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event raise exc_type(*result[2]) 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b. 2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event The thing is this line of code should actually be creating the new namespace[2] so not sure why its complaining that namespace was not found. I am suspecting there is some race condition or more likely some test interferance due to test runner concurrency. [0] https://zuul.opendev.org/t/openstack/build/5d1910037db844e88cf9ef694068cf17 [1] https://review.opendev.org/c/openstack/neutron/+/875586 [2] https://github.com/openstack/neutron/blob/master/neutron/agent/ovn/metadata/agent.py#LL610-L611 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2011377/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
