Public bug reported: test_create_router_set_gateway_with_fixed_ip periodically fails in neutron-tempest-dvr-ha-multinode-full. Failure start to happen after new engine facade switch in L3 code. Failure is due to router failed to be deleted (neutron returns 500), see first line in below traceback.
Traceback: 2021-01-28 13:36:26,139 81948 INFO [tempest.lib.common.rest_client] Request (RoutersAdminTest:test_create_router_set_gateway_with_fixed_ip): 500 DELETE https://10.209.160.170:9696/v2.0/routers/3f337e0e-3bed-44f7-a8f9-e43c01787445 10.779s 2021-01-28 13:36:26,139 81948 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'} Body: None Response - Headers: {'date': 'Thu, 28 Jan 2021 13:36:26 GMT', 'server': 'Apache/2.4.41 (Ubuntu)', 'content-type': 'application/json', 'content-length': '150', 'x-openstack-request-id': 'req-2392442f-a3b0-4c79-a79c-c765c7cab834', 'connection': 'close', 'status': '500', 'content-location': 'https://10.209.160.170:9696/v2.0/routers/3f337e0e-3bed-44f7-a8f9-e43c01787445'} Body: b'{"NeutronError": {"type": "HTTPInternalServerError", "message": "Request Failed: internal server error while processing your request.", "detail": ""}}' 2021-01-28 13:36:26,305 81948 INFO [tempest.lib.common.rest_client] Request (RoutersAdminTest:_run_cleanups): 409 DELETE https://10.209.160.170:9696/v2.0/subnets/812a9855-15a2-4a8e-b246-c6ea68cdadcd 0.165s 2021-01-28 13:36:26,306 81948 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'} Body: None Response - Headers: {'date': 'Thu, 28 Jan 2021 13:36:26 GMT', 'server': 'Apache/2.4.41 (Ubuntu)', 'content-type': 'application/json', 'content-length': '204', 'x-openstack-request-id': 'req-9eb0af8b-9154-489b-94b8-93d3da458c05', 'connection': 'close', 'status': '409', 'content-location': 'https://10.209.160.170:9696/v2.0/subnets/812a9855-15a2-4a8e-b246-c6ea68cdadcd'} Body: b'{"NeutronError": {"type": "SubnetInUse", "message": "Unable to complete operation on subnet 812a9855-15a2-4a8e-b246-c6ea68cdadcd: One or more ports have an IP allocation from this subnet.", "detail": ""}}' 2021-01-28 13:36:26,544 81948 INFO [tempest.lib.common.rest_client] Request (RoutersAdminTest:_run_cleanups): 409 DELETE https://10.209.160.170:9696/v2.0/networks/c06258a3-d817-4ffd-b9c6-1c20eaedd688 0.238s 2021-01-28 13:36:26,545 81948 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'} Body: None Response - Headers: {'date': 'Thu, 28 Jan 2021 13:36:26 GMT', 'server': 'Apache/2.4.41 (Ubuntu)', 'content-type': 'application/json', 'content-length': '205', 'x-openstack-request-id': 'req-a412a0f0-832c-45f4-9e9d-40deef43cde2', 'connection': 'close', 'status': '409', 'content-location': 'https://10.209.160.170:9696/v2.0/networks/c06258a3-d817-4ffd-b9c6-1c20eaedd688'} Body: b'{"NeutronError": {"type": "NetworkInUse", "message": "Unable to complete operation on network c06258a3-d817-4ffd-b9c6-1c20eaedd688. There are one or more ports still in use on the network.", "detail": ""}}' }}} traceback-1: {{{ Traceback (most recent call last): File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 87, in call_and_ignore_notfound_exc return func(*args, **kwargs) File "/opt/stack/tempest/tempest/lib/services/network/subnets_client.py", line 52, in delete_subnet return self.delete_resource(uri) File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource resp, body = self.delete(req_uri) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 331, in delete return self.request('DELETE', url, extra_headers, headers, body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 704, in request self._error_checker(resp, resp_body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 825, in _error_checker raise exceptions.Conflict(resp_body, resp=resp) tempest.lib.exceptions.Conflict: Conflict with state of target resource Details: {'type': 'SubnetInUse', 'message': 'Unable to complete operation on subnet 812a9855-15a2-4a8e-b246-c6ea68cdadcd: One or more ports have an IP allocation from this subnet.', 'detail': ''} }}} traceback-2: {{{ Traceback (most recent call last): File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 87, in call_and_ignore_notfound_exc return func(*args, **kwargs) File "/opt/stack/tempest/tempest/lib/services/network/networks_client.py", line 52, in delete_network return self.delete_resource(uri) File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource resp, body = self.delete(req_uri) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 331, in delete return self.request('DELETE', url, extra_headers, headers, body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 704, in request self._error_checker(resp, resp_body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 825, in _error_checker raise exceptions.Conflict(resp_body, resp=resp) tempest.lib.exceptions.Conflict: Conflict with state of target resource Details: {'type': 'NetworkInUse', 'message': 'Unable to complete operation on network c06258a3-d817-4ffd-b9c6-1c20eaedd688. There are one or more ports still in use on the network.', 'detail': ''} }}} Traceback (most recent call last): File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 109, in wrapper return func(*func_args, **func_kwargs) File "/opt/stack/tempest/tempest/api/network/admin/test_routers.py", line 254, in test_create_router_set_gateway_with_fixed_ip self.admin_routers_client.delete_router(router['id']) File "/opt/stack/tempest/tempest/lib/services/network/routers_client.py", line 52, in delete_router return self.delete_resource(uri) File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource resp, body = self.delete(req_uri) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 331, in delete return self.request('DELETE', url, extra_headers, headers, body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 704, in request self._error_checker(resp, resp_body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 880, in _error_checker raise exceptions.ServerFault(resp_body, resp=resp, tempest.lib.exceptions.ServerFault: Got server fault Details: Request Failed: internal server error while processing your request. ** Affects: neutron Importance: High Assignee: Oleg Bondarev (obondarev) Status: Confirmed ** Tags: gate-failure l3-dvr-backlog -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1915271 Title: test_create_router_set_gateway_with_fixed_ip fails in dvr-ha job Status in neutron: Confirmed Bug description: test_create_router_set_gateway_with_fixed_ip periodically fails in neutron-tempest-dvr-ha-multinode-full. Failure start to happen after new engine facade switch in L3 code. Failure is due to router failed to be deleted (neutron returns 500), see first line in below traceback. Traceback: 2021-01-28 13:36:26,139 81948 INFO [tempest.lib.common.rest_client] Request (RoutersAdminTest:test_create_router_set_gateway_with_fixed_ip): 500 DELETE https://10.209.160.170:9696/v2.0/routers/3f337e0e-3bed-44f7-a8f9-e43c01787445 10.779s 2021-01-28 13:36:26,139 81948 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'} Body: None Response - Headers: {'date': 'Thu, 28 Jan 2021 13:36:26 GMT', 'server': 'Apache/2.4.41 (Ubuntu)', 'content-type': 'application/json', 'content-length': '150', 'x-openstack-request-id': 'req-2392442f-a3b0-4c79-a79c-c765c7cab834', 'connection': 'close', 'status': '500', 'content-location': 'https://10.209.160.170:9696/v2.0/routers/3f337e0e-3bed-44f7-a8f9-e43c01787445'} Body: b'{"NeutronError": {"type": "HTTPInternalServerError", "message": "Request Failed: internal server error while processing your request.", "detail": ""}}' 2021-01-28 13:36:26,305 81948 INFO [tempest.lib.common.rest_client] Request (RoutersAdminTest:_run_cleanups): 409 DELETE https://10.209.160.170:9696/v2.0/subnets/812a9855-15a2-4a8e-b246-c6ea68cdadcd 0.165s 2021-01-28 13:36:26,306 81948 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'} Body: None Response - Headers: {'date': 'Thu, 28 Jan 2021 13:36:26 GMT', 'server': 'Apache/2.4.41 (Ubuntu)', 'content-type': 'application/json', 'content-length': '204', 'x-openstack-request-id': 'req-9eb0af8b-9154-489b-94b8-93d3da458c05', 'connection': 'close', 'status': '409', 'content-location': 'https://10.209.160.170:9696/v2.0/subnets/812a9855-15a2-4a8e-b246-c6ea68cdadcd'} Body: b'{"NeutronError": {"type": "SubnetInUse", "message": "Unable to complete operation on subnet 812a9855-15a2-4a8e-b246-c6ea68cdadcd: One or more ports have an IP allocation from this subnet.", "detail": ""}}' 2021-01-28 13:36:26,544 81948 INFO [tempest.lib.common.rest_client] Request (RoutersAdminTest:_run_cleanups): 409 DELETE https://10.209.160.170:9696/v2.0/networks/c06258a3-d817-4ffd-b9c6-1c20eaedd688 0.238s 2021-01-28 13:36:26,545 81948 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'} Body: None Response - Headers: {'date': 'Thu, 28 Jan 2021 13:36:26 GMT', 'server': 'Apache/2.4.41 (Ubuntu)', 'content-type': 'application/json', 'content-length': '205', 'x-openstack-request-id': 'req-a412a0f0-832c-45f4-9e9d-40deef43cde2', 'connection': 'close', 'status': '409', 'content-location': 'https://10.209.160.170:9696/v2.0/networks/c06258a3-d817-4ffd-b9c6-1c20eaedd688'} Body: b'{"NeutronError": {"type": "NetworkInUse", "message": "Unable to complete operation on network c06258a3-d817-4ffd-b9c6-1c20eaedd688. There are one or more ports still in use on the network.", "detail": ""}}' }}} traceback-1: {{{ Traceback (most recent call last): File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 87, in call_and_ignore_notfound_exc return func(*args, **kwargs) File "/opt/stack/tempest/tempest/lib/services/network/subnets_client.py", line 52, in delete_subnet return self.delete_resource(uri) File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource resp, body = self.delete(req_uri) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 331, in delete return self.request('DELETE', url, extra_headers, headers, body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 704, in request self._error_checker(resp, resp_body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 825, in _error_checker raise exceptions.Conflict(resp_body, resp=resp) tempest.lib.exceptions.Conflict: Conflict with state of target resource Details: {'type': 'SubnetInUse', 'message': 'Unable to complete operation on subnet 812a9855-15a2-4a8e-b246-c6ea68cdadcd: One or more ports have an IP allocation from this subnet.', 'detail': ''} }}} traceback-2: {{{ Traceback (most recent call last): File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 87, in call_and_ignore_notfound_exc return func(*args, **kwargs) File "/opt/stack/tempest/tempest/lib/services/network/networks_client.py", line 52, in delete_network return self.delete_resource(uri) File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource resp, body = self.delete(req_uri) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 331, in delete return self.request('DELETE', url, extra_headers, headers, body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 704, in request self._error_checker(resp, resp_body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 825, in _error_checker raise exceptions.Conflict(resp_body, resp=resp) tempest.lib.exceptions.Conflict: Conflict with state of target resource Details: {'type': 'NetworkInUse', 'message': 'Unable to complete operation on network c06258a3-d817-4ffd-b9c6-1c20eaedd688. There are one or more ports still in use on the network.', 'detail': ''} }}} Traceback (most recent call last): File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 109, in wrapper return func(*func_args, **func_kwargs) File "/opt/stack/tempest/tempest/api/network/admin/test_routers.py", line 254, in test_create_router_set_gateway_with_fixed_ip self.admin_routers_client.delete_router(router['id']) File "/opt/stack/tempest/tempest/lib/services/network/routers_client.py", line 52, in delete_router return self.delete_resource(uri) File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 41, in delete_resource resp, body = self.delete(req_uri) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 331, in delete return self.request('DELETE', url, extra_headers, headers, body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 704, in request self._error_checker(resp, resp_body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 880, in _error_checker raise exceptions.ServerFault(resp_body, resp=resp, tempest.lib.exceptions.ServerFault: Got server fault Details: Request Failed: internal server error while processing your request. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1915271/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
