Reviewed: https://review.opendev.org/c/openstack/neutron/+/877535
Committed:
https://opendev.org/openstack/neutron/commit/04d3f889efeef74e8eb8d8bf330f1594913b161a
Submitter: "Zuul (22348)"
Branch: master
commit 04d3f889efeef74e8eb8d8bf330f1594913b161a
Author: Miro Tomaska <mtoma...@redhat.com>
Date: Wed Mar 15 12:26:12 2023 -0500
Fix metadata agent intermittent test failures
Metadata agent has been experiencing intermittent failures
mostly because of test conccurency and how the metadata agent
code assumes its the only process running on the system and
operating on the ovnmeta-* namespaces. See comment#1 the
linked bug for more details. Although I dont like forcing
--concurrency 1 for this test class, I think that is going
to be the best solution and any new tests that will be added
in the future.
Closes-Bug: #2011377
Change-Id: Ie7f3b496de6b23be5739fbeba10f53602e8b300d
** Changed in: neutron
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2011377
Title:
test_agent_resync_on_non_existing_bridge failing intermittently sp
Status in neutron:
Fix Released
Bug description:
Test
neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent.test_agent_resync_on_non_existing_bridge
is failing intermitelly. Ex [0]
I just reintroduced this test [1] into the code. The failure is does
not happen all the time but I can be reproduce it locally with
--until-failure with multiple concurrency(big hint) and running the
whole TestMetadaAgent class of tests (another hint). Like this
`tox -e dsvm-functional --
neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent
--until-failure --concurrency 0`
When the failure happens following exception is found in the logs
2023-03-10 17:49:11.861 40848 INFO neutron.agent.ovn.metadata.agent [-] Port
ovn-port-feb89eb1-fcf8-4b38-8aee-d8dd4b6b497e in datapath
ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b bound to our chassis
2023-03-10 17:49:11.863 40848 INFO neutron.agent.ovn.metadata.agent [-]
Provisioning metadata for network ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b
2023-03-10 17:49:11.917 40848 DEBUG neutron.agent.ovn.metadata.agent [-]
Creating VETH tapovn-f376a61 in
ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b namespace provision_datapath
/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py:603
2023-03-10 17:49:11.923 41596 DEBUG neutron.privileged.agent.linux.ip_lib [-]
Interface tapovn-f376a60 not found in namespace None get_link_id
/home/zuul/src/opendev.org/openstack/neutron/neutron/privileged/agent/linux/ip_lib.py:204
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event [-] Unexpected exception
in notify_loop: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound:
Network interface tapovn-f376a61 not found in namespace
ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b.
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event Traceback (most recent
call last):
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/event.py",
line 177, in notify_loop
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event match.run(event, row,
updates)
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py",
line 110, in run
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event
self.agent.provision_datapath(row.datapath)
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py",
line 640, in provision_datapath
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event
ip2.addr.add_multiple(ipv4_cidrs_to_add)
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py",
line 544, in add_multiple
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event
add_ip_addresses(cidrs, self.name, self._parent.namespace, scope,
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py",
line 848, in add_ip_addresses
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event
privileged.add_ip_addresses(
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/priv_context.py",
line 271, in _wrap
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event return
self.channel.remote_call(name, args, kwargs,
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/daemon.py",
line 215, in remote_call
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event raise
exc_type(*result[2])
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event
neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network
interface tapovn-f376a61 not found in namespace
ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b.
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event
The thing is this line of code should actually be creating the new
namespace[2] so not sure why its complaining that namespace was not
found. I am suspecting there is some race condition or more likely
some test interferance due to test runner concurrency.
[0]
https://zuul.opendev.org/t/openstack/build/5d1910037db844e88cf9ef694068cf17
[1] https://review.opendev.org/c/openstack/neutron/+/875586
[2]
https://github.com/openstack/neutron/blob/master/neutron/agent/ovn/metadata/agent.py#LL610-L611
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2011377/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
