Public bug reported: As shown in http://logs.openstack.org/71/520371/7/check/legacy-neutron- dsvm-fullstack/ad585a2/logs/dsvm-fullstack- logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow- native_.txt.gz fullstack doesn't wait enough time for agents to report to neutron server. Currently it uses default value of 60 seconds, so once all agent processes are started they have 60 seconds to report back.
On busy or weak hardware machines this is not enough. Here comes RCA: Logs: Problematic neutron-openvswitch-agent: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-openvswitch-agent--2017-11-20--21-51-57-666771.txt.gz Neutron-server: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-server--2017-11-20--21-51-23-097309.txt.gz Test runner: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz RCA: Agent started by test runner at 21:51:57.666: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz#_2017-11-20_21_51_57_666 Test runner stopped polling server for agents at 21:52:51.506: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz#_2017-11-20_21_52_51_506 Agent process started at 21:52:23.444, 26 seconds! after async_process claimed process has started Server got report of the agent at 21:52:58.939 - 7 seconds after test runner stopped polling: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-server--2017-11-20--21-51-23-097309.txt.gz#_2017-11-20_21_52_58_939 Summary: On environment with three nodes, running dhcp and ovs agent on each, last agent reported 7 seconds after API polling was finished by test runner. ** Affects: neutron Importance: Undecided Assignee: Jakub Libosvar (libosvar) Status: New ** Tags: fullstack -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1734356 Title: fullstack: Test runner doesn't wait enough time for env to come up Status in neutron: New Bug description: As shown in http://logs.openstack.org/71/520371/7/check/legacy- neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack- logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow- native_.txt.gz fullstack doesn't wait enough time for agents to report to neutron server. Currently it uses default value of 60 seconds, so once all agent processes are started they have 60 seconds to report back. On busy or weak hardware machines this is not enough. Here comes RCA: Logs: Problematic neutron-openvswitch-agent: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-openvswitch-agent--2017-11-20--21-51-57-666771.txt.gz Neutron-server: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-server--2017-11-20--21-51-23-097309.txt.gz Test runner: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz RCA: Agent started by test runner at 21:51:57.666: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz#_2017-11-20_21_51_57_666 Test runner stopped polling server for agents at 21:52:51.506: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_.txt.gz#_2017-11-20_21_52_51_506 Agent process started at 21:52:23.444, 26 seconds! after async_process claimed process has started Server got report of the agent at 21:52:58.939 - 7 seconds after test runner stopped polling: http://logs.openstack.org/71/520371/7/check/legacy-neutron-dsvm-fullstack/ad585a2/logs/dsvm-fullstack-logs/TestOvsConnectivitySameNetwork.test_connectivity_VXLAN,openflow-native_/neutron-server--2017-11-20--21-51-23-097309.txt.gz#_2017-11-20_21_52_58_939 Summary: On environment with three nodes, running dhcp and ovs agent on each, last agent reported 7 seconds after API polling was finished by test runner. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1734356/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
