Public bug reported: Neutron return 500 when attempting to change name of an existing router:
$ openstack router set --name router-flat 29ead671-3957-4bf2-9dbf-e1bff0a12ab2 HttpException: 500: Server Error for url: http://192.168.121.189:9696/v2.0/routers/29ead671-3957-4bf2-9dbf-e1bff0a12ab2, Request Failed: internal server error while processing your request. May 21 13:29:47 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-696e578a-91c1-4f9b-8a74-2e17596b6bda None None] Running txn n=1 command(idx=0): DbSetCommand(table=Chassis, record=141047c5-57f5-443a-a50f-db65612c047b, col_values=(('external_ids', {'neutron:liveness_check_at': '2020-05-21T13:29:47.048155+00:00'}),)) {{(pid=3400) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-696e578a-91c1-4f9b-8a74-2e17596b6bda None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster May 21 13:29:47 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-696e578a-91c1-4f9b-8a74-2e17596b6bda None None] Running txn n=1 command(idx=0): DbSetCommand(table=Chassis, record=141047c5-57f5-443a-a50f-db65612c047b, col_values=(('external_ids', {'neutron:metadata_liveness_check_at': '2020-05-21T13:29:47.055462+00:00'}),)) {{(pid=3400) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-f37ed5e0-ef6a-4fc0-9670-537d4935f18e None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-cb4458e4-ff28-4d3d-ad22-21e13625998f None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-cb4458e4-ff28-4d3d-ad22-21e13625998f None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-696e578a-91c1-4f9b-8a74-2e17596b6bda None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-f37ed5e0-ef6a-4fc0-9670-537d4935f18e None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster May 21 13:29:47 devstack neutron-server[3378]: DEBUG neutron.db.agents_db [None req-29c44001-08f7-4ff1-9885-2a8fc4dc7e61 None None] Agent healthcheck: found 2 active agents {{(pid=3400) agent_health_check /opt/stack/neutron/neutron/db/agents_db.py:327}} May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-38aed43c-4b9a-4521-9e69-349701e72f62 service neutron] 192.168.121.189 "GET /v2.0/ports?device_id=aff97800-74d0-4339-83d5-3ae53bbc9d97&fields=binding%3Ahost_id&fields=binding%3Avif_type HTTP/1.1" status: 200 len: 265 time: 0.0434918 May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-d77dcd0a-bac0-49ec-ba88-73e2a220cb6c service neutron] 192.168.121.189 "GET /v2.0/ports?tenant_id=9635d9bacb954ae78b28222e6eba4070&device_id=aff97800-74d0-4339-83d5-3ae53bbc9d97 HTTP/1.1" status: 200 len: 1456 time: 0.0404735 May 21 13:29:48 devstack neutron-server[3378]: DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-9312b103-df29-4f4c-9ba6-93dbf47080af service neutron] Attributes excluded by policy engine: ['vlan_transparent'] {{(pid=3395) _exclude_attributes_by_policy /opt/stack/neutron/neutron/pecan_wsgi/hooks/policy_enforcement.py:256}} May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-9312b103-df29-4f4c-9ba6-93dbf47080af service neutron] 192.168.121.189 "GET /v2.0/networks?id=336d1bc4-69c6-4b70-b48c-36b190d979db HTTP/1.1" status: 200 len: 930 time: 0.0695074 May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-48e27f2d-f333-4b8a-b992-fa46d6a849ba service neutron] 192.168.121.189 "GET /v2.0/floatingips?fixed_ip_address=10.0.0.2&port_id=3d0ce33c-fcf8-4073-95b5-ebf4c5722c74 HTTP/1.1" status: 200 len: 1032 time: 0.0478008 May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-2354eb84-e80c-4abb-91f3-25701844207b service neutron] 192.168.121.189 "GET /v2.0/floatingips?fixed_ip_address=fd4f%3A707c%3A5223%3A0%3Af816%3A3eff%3Afeac%3Aa720&port_id=3d0ce33c-fcf8-4073-95b5-ebf4c5722c74 HTTP/1.1" status: 200 len: 217 time: 0.0307662 May 21 13:29:48 devstack neutron-server[3378]: DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-8440b651-4880-49ed-8b5d-014790d07f34 service neutron] Attributes excluded by policy engine: ['shared'] {{(pid=3395) _exclude_attributes_by_policy /opt/stack/neutron/neutron/pecan_wsgi/hooks/policy_enforcement.py:256}} May 21 13:29:48 devstack neutron-server[3378]: DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-8440b651-4880-49ed-8b5d-014790d07f34 service neutron] Attributes excluded by policy engine: ['shared'] {{(pid=3395) _exclude_attributes_by_policy /opt/stack/neutron/neutron/pecan_wsgi/hooks/policy_enforcement.py:256}} May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-8440b651-4880-49ed-8b5d-014790d07f34 service neutron] 192.168.121.189 "GET /v2.0/subnets?id=5cce10c1-b3c3-4e3f-b709-2066dcc2c785&id=28dfb2ed-2662-4638-a2d4-9369c64b2f3c HTTP/1.1" status: 200 len: 1599 time: 0.0369835 May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-a6779a14-962e-461e-b806-be974da65e35 service neutron] 192.168.121.189 "GET /v2.0/ports?network_id=336d1bc4-69c6-4b70-b48c-36b190d979db&device_owner=network%3Adhcp HTTP/1.1" status: 200 len: 1441 time: 0.0365410 May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-7db1d18d-7e32-4a44-a9e8-e855fd16a079 service neutron] 192.168.121.189 "GET /v2.0/ports?network_id=336d1bc4-69c6-4b70-b48c-36b190d979db&device_owner=network%3Adhcp HTTP/1.1" status: 200 len: 1441 time: 0.0359824 May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-8a3492de-0578-4c0b-bab6-4bf86f630196 service neutron] 192.168.121.189 "GET /v2.0/networks/336d1bc4-69c6-4b70-b48c-36b190d979db?fields=segments HTTP/1.1" status: 200 len: 212 time: 0.0682638 May 21 13:29:49 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-211dc48a-64e4-4e49-b630-88d45c543861 service neutron] 192.168.121.189 "GET /v2.0/networks/336d1bc4-69c6-4b70-b48c-36b190d979db?fields=provider%3Aphysical_network&fields=provider%3Anetwork_type HTTP/1.1" status: 200 len: 277 time: 0.0675299 May 21 13:29:56 devstack neutron-server[3378]: DEBUG neutron.wsgi [-] (3395) accepted ('192.168.121.189', 54086) {{(pid=3395) server /usr/local/lib/python3.6/site-packages/eventlet/wsgi.py:985}} May 21 13:29:56 devstack neutron-server[3378]: INFO neutron.api.v2.resource [None req-c451591f-e2b5-45d3-964e-e26a45f847bb demo admin] show failed (client error): The resource could not be found. May 21 13:29:56 devstack neutron-server[3378]: INFO neutron.wsgi [None req-c451591f-e2b5-45d3-964e-e26a45f847bb demo admin] 192.168.121.189 "GET /v2.0/routers/router1 HTTP/1.1" status: 404 len: 312 time: 0.0663443 May 21 13:29:57 devstack neutron-server[3378]: INFO neutron.wsgi [None req-82148186-c9ef-4e98-977c-0efbfeafbf84 demo admin] 192.168.121.189 "GET /v2.0/routers?name=router1 HTTP/1.1" status: 200 len: 859 time: 0.1023729 May 21 13:29:57 devstack neutron-server[3378]: DEBUG neutron.api.v2.base [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Request body: {'router': {'name': 'router-flat'}} {{(pid=3395) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:719}} May 21 13:29:57 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, precommit_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}} May 21 13:29:57 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, after_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Chassis candidates with external connectivity: ['30a0cb52-356d-4f2d-be32-821652201fc7'] {{(pid=3395) get_candidates_for_scheduling /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py:1412}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron.scheduler.l3_ovn_scheduler [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Gateway lrp-d7c38c52-9350-47bf-9275-118a24538a9f scheduled on chassis ['30a0cb52-356d-4f2d-be32-821652201fc7'] {{(pid=3395) _schedule_gateway /opt/stack/neutron/neutron/scheduler/l3_ovn_scheduler.py:85}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(name=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, resource={'id': '29ead671-3957-4bf2-9dbf-e1bff0a12ab2', 'name': 'router-flat', 'tenant_id': '9635d9bacb954ae78b28222e6eba4070', 'admin_state_up': True, 'status': 'ACTIVE', 'external_gateway_info': {'network_id': 'c1b6e05b-b622-4565-8a46-c03cd25b7c4b', 'external_fixed_ips': [{'subnet_id': 'ed6092d5-35d2-4dce-b347-dfeba6a5dd7f', 'ip_address': '172.24.4.51'}, {'subnet_id': 'c2e143e5-8f48-45e0-b23e-6bb497381e7b', 'ip_address': '2001:db8::1'}], 'enable_snat': True}, 'gw_port_id': 'd7c38c52-9350-47bf-9275-118a24538a9f', 'description': '', 'routes': [], 'tags': [], 'created_at': '2020-05-20T11:27:32Z', 'updated_at': '2020-05-21T13:29:57Z', 'revision_number': 9, 'project_id': '9635d9bacb954ae78b28222e6eba4070'}, resource_type=routers, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DeleteLRouterExtGwCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DeleteNATRuleInLRouterCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, type=snat, logical_ip=10.0.0.0/26, external_ip=172.24.4.51, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, ip_prefix=0.0.0.0/0, nexthop=172.24.4.1, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, ip_prefix=::/0, nexthop=2001:db8::2, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelLRouterPortCommand(name=lrp-d7c38c52-9350-47bf-9275-118a24538a9f, lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): AddLRouterPortCommand(name=lrp-d7c38c52-9350-47bf-9275-118a24538a9f, lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, may_exist=True, columns={'mac': 'fa:16:3e:54:cf:1a', 'networks': ['2001:db8::1/64', '172.24.4.51/24'], 'external_ids': {'neutron:revision_number': '6', 'neutron:subnet_ids': 'ed6092d5-35d2-4dce-b347-dfeba6a5dd7f c2e143e5-8f48-45e0-b23e-6bb497381e7b', 'neutron:network_name': 'neutron-c1b6e05b-b622-4565-8a46-c03cd25b7c4b', 'neutron:router_name': '29ead671-3957-4bf2-9dbf-e1bff0a12ab2'}, 'options': {}, 'gateway_chassis': ['30a0cb52-356d-4f2d-be32-821652201fc7']}) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): SetLRouterPortInLSwitchPortCommand(lswitch_port=d7c38c52-9350-47bf-9275-118a24538a9f, lrouter_port=lrp-d7c38c52-9350-47bf-9275-118a24538a9f, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): AddStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, columns={'ip_prefix': '0.0.0.0/0', 'nexthop': '172.24.4.1', 'external_ids': {'neutron:is_ext_gw': 'true', 'neutron:subnet_id': 'ed6092d5-35d2-4dce-b347-dfeba6a5dd7f'}}) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): AddStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, columns={'ip_prefix': '::/0', 'nexthop': '2001:db8::2', 'external_ids': {'neutron:is_ext_gw': 'true', 'neutron:subnet_id': 'c2e143e5-8f48-45e0-b23e-6bb497381e7b'}}) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): AddNATRuleInLRouterCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, columns={'type': 'snat', 'logical_ip': '10.0.0.0/26', 'external_ip': '172.24.4.51'}) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): UpdateLRouterCommand(name=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, columns={'external_ids': {'neutron:router_name': 'router-flat', 'neutron:gw_port_id': 'd7c38c52-9350-47bf-9275-118a24538a9f', 'neutron:revision_number': '9'}, 'enabled': True}, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, ip_prefix=0.0.0.0/0, nexthop=172.24.4.1, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, ip_prefix=::/0, nexthop=2001:db8::2, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"} May 21 13:29:58 devstack neutron-server[3378]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Traceback (most recent call last): May 21 13:29:58 devstack neutron-server[3378]: File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run May 21 13:29:58 devstack neutron-server[3378]: txn.results.put(txn.do_commit()) May 21 13:29:58 devstack neutron-server[3378]: File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit May 21 13:29:58 devstack neutron-server[3378]: raise RuntimeError(msg) May 21 13:29:58 devstack neutron-server[3378]: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"} May 21 13:29:58 devstack neutron-server[3378]: May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Unable to update router 29ead671-3957-4bf2-9dbf-e1bff0a12ab2. Error: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"} May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Unable to update lrouter for 29ead671-3957-4bf2-9dbf-e1bff0a12ab2: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"} May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin Traceback (most recent call last): May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 142, in update_router May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin original_router) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1374, in update_router May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin 'error': e}) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin self.force_reraise() May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin six.reraise(self.type_, self.value, self.tb) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin raise value May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1356, in update_router May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin context, router_id, added, removed, txn=txn) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin next(self.gen) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 185, in transaction May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin yield t May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin next(self.gen) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin del self._nested_txns_map[cur_thread_id] May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin self.result = self.commit() May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin raise result.ex May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin txn.results.put(txn.do_commit()) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin raise RuntimeError(msg) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"} May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron.db.extraroute_db [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Added routes are [] {{(pid=3395) _update_extra_routes /opt/stack/neutron/neutron/db/extraroute_db.py:113}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron.db.extraroute_db [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Removed routes are [] {{(pid=3395) _update_extra_routes /opt/stack/neutron/neutron/db/extraroute_db.py:121}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, precommit_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, precommit_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, precommit_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}} May 21 13:29:58 devstack neutron-server[3378]: ERROR root [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 142, in update_router\n original_router)\n', ' File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1374, in update_router\n \'error\': e})\n', ' File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1356, in update_router\n context, router_id, added, removed, txn=txn)\n', ' File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__\n next(self.gen)\n', ' File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 185, in transaction\n yield t\n', ' File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__\n next(self.gen)\n', ' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction\n del self._nested_txns_map[cur_thread_id]\n', ' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__\n self.result = self.commit()\n', ' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit\n raise result.ex\n', ' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run\n txn.results.put(txn.do_commit())\n', ' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit\n raise RuntimeError(msg)\n', 'RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}\n']: sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <Router at 0x7ff80bfad400> is not bound to a Session; lazy load operation of attribute 'gw_port' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] update failed: No details.: sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <Router at 0x7ff80bfad400> is not bound to a Session; lazy load operation of attribute 'gw_port' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource Traceback (most recent call last): May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 142, in update_router May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource original_router) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1374, in update_router May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource 'error': e}) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise() May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1356, in update_router May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource context, router_id, added, removed, txn=txn) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource next(self.gen) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 185, in transaction May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource yield t May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource next(self.gen) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource del self._nested_txns_map[cur_thread_id] May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.result = self.commit() May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise result.ex May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource txn.results.put(txn.do_commit()) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise RuntimeError(msg) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"} May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource During handling of the above exception, another exception occurred: May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource Traceback (most recent call last): May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/resource.py", line 98, in resource May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource result = method(request=request, **args) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 625, in update May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return self._update(request, id, body, **kwargs) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise() May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return f(*args, **kwargs) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource ectxt.value = e.inner_exc May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise() May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return f(*args, **kwargs) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise() May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 685, in _update May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource obj = obj_updater(request.context, id, **kwargs) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 148, in update_router May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource revert_router) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/extraroute_db.py", line 72, in update_router May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource context, id, router) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/l3_db.py", line 1963, in update_router May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource id, router) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return method(*args, **kwargs) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise() May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return f(*args, **kwargs) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource ectxt.value = e.inner_exc May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise() May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return f(*args, **kwargs) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise() May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/l3_db.py", line 287, in update_router May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource updated = self._make_router_dict(router_db) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/l3_db.py", line 198, in _make_router_dict May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource 'network_id': router.gw_port['network_id'], May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 287, in __get__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return self.impl.get(instance_state(instance), dict_) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 723, in get May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource value = self.callable_(state, passive) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 710, in _load_for_state May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource % (orm_util.state_str(state), self.key) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <Router at 0x7ff80bfad400> is not bound to a Session; lazy load operation of attribute 'gw_port' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource May 21 13:29:58 devstack neutron-server[3378]: INFO neutron.wsgi [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] 192.168.121.189 "PUT /v2.0/routers/29ead671-3957-4bf2-9dbf-e1bff0a12ab2 HTTP/1.1" status: 500 len: 368 time: 1.8027527 ** Affects: neutron Importance: Undecided Status: New ** Tags: ovn -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1879950 Title: Unable to update router name with ovn Status in neutron: New Bug description: Neutron return 500 when attempting to change name of an existing router: $ openstack router set --name router-flat 29ead671-3957-4bf2-9dbf-e1bff0a12ab2 HttpException: 500: Server Error for url: http://192.168.121.189:9696/v2.0/routers/29ead671-3957-4bf2-9dbf-e1bff0a12ab2, Request Failed: internal server error while processing your request. May 21 13:29:47 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-696e578a-91c1-4f9b-8a74-2e17596b6bda None None] Running txn n=1 command(idx=0): DbSetCommand(table=Chassis, record=141047c5-57f5-443a-a50f-db65612c047b, col_values=(('external_ids', {'neutron:liveness_check_at': '2020-05-21T13:29:47.048155+00:00'}),)) {{(pid=3400) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-696e578a-91c1-4f9b-8a74-2e17596b6bda None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster May 21 13:29:47 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-696e578a-91c1-4f9b-8a74-2e17596b6bda None None] Running txn n=1 command(idx=0): DbSetCommand(table=Chassis, record=141047c5-57f5-443a-a50f-db65612c047b, col_values=(('external_ids', {'neutron:metadata_liveness_check_at': '2020-05-21T13:29:47.055462+00:00'}),)) {{(pid=3400) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-f37ed5e0-ef6a-4fc0-9670-537d4935f18e None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-cb4458e4-ff28-4d3d-ad22-21e13625998f None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-cb4458e4-ff28-4d3d-ad22-21e13625998f None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-696e578a-91c1-4f9b-8a74-2e17596b6bda None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster May 21 13:29:47 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [None req-f37ed5e0-ef6a-4fc0-9670-537d4935f18e None None] HashRing is empty, error: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'141047c5-57f5-443a-a50f-db65612c047b'". This should never happen in a normal situation, please check the status of your cluster May 21 13:29:47 devstack neutron-server[3378]: DEBUG neutron.db.agents_db [None req-29c44001-08f7-4ff1-9885-2a8fc4dc7e61 None None] Agent healthcheck: found 2 active agents {{(pid=3400) agent_health_check /opt/stack/neutron/neutron/db/agents_db.py:327}} May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-38aed43c-4b9a-4521-9e69-349701e72f62 service neutron] 192.168.121.189 "GET /v2.0/ports?device_id=aff97800-74d0-4339-83d5-3ae53bbc9d97&fields=binding%3Ahost_id&fields=binding%3Avif_type HTTP/1.1" status: 200 len: 265 time: 0.0434918 May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-d77dcd0a-bac0-49ec-ba88-73e2a220cb6c service neutron] 192.168.121.189 "GET /v2.0/ports?tenant_id=9635d9bacb954ae78b28222e6eba4070&device_id=aff97800-74d0-4339-83d5-3ae53bbc9d97 HTTP/1.1" status: 200 len: 1456 time: 0.0404735 May 21 13:29:48 devstack neutron-server[3378]: DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-9312b103-df29-4f4c-9ba6-93dbf47080af service neutron] Attributes excluded by policy engine: ['vlan_transparent'] {{(pid=3395) _exclude_attributes_by_policy /opt/stack/neutron/neutron/pecan_wsgi/hooks/policy_enforcement.py:256}} May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-9312b103-df29-4f4c-9ba6-93dbf47080af service neutron] 192.168.121.189 "GET /v2.0/networks?id=336d1bc4-69c6-4b70-b48c-36b190d979db HTTP/1.1" status: 200 len: 930 time: 0.0695074 May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-48e27f2d-f333-4b8a-b992-fa46d6a849ba service neutron] 192.168.121.189 "GET /v2.0/floatingips?fixed_ip_address=10.0.0.2&port_id=3d0ce33c-fcf8-4073-95b5-ebf4c5722c74 HTTP/1.1" status: 200 len: 1032 time: 0.0478008 May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-2354eb84-e80c-4abb-91f3-25701844207b service neutron] 192.168.121.189 "GET /v2.0/floatingips?fixed_ip_address=fd4f%3A707c%3A5223%3A0%3Af816%3A3eff%3Afeac%3Aa720&port_id=3d0ce33c-fcf8-4073-95b5-ebf4c5722c74 HTTP/1.1" status: 200 len: 217 time: 0.0307662 May 21 13:29:48 devstack neutron-server[3378]: DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-8440b651-4880-49ed-8b5d-014790d07f34 service neutron] Attributes excluded by policy engine: ['shared'] {{(pid=3395) _exclude_attributes_by_policy /opt/stack/neutron/neutron/pecan_wsgi/hooks/policy_enforcement.py:256}} May 21 13:29:48 devstack neutron-server[3378]: DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-8440b651-4880-49ed-8b5d-014790d07f34 service neutron] Attributes excluded by policy engine: ['shared'] {{(pid=3395) _exclude_attributes_by_policy /opt/stack/neutron/neutron/pecan_wsgi/hooks/policy_enforcement.py:256}} May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-8440b651-4880-49ed-8b5d-014790d07f34 service neutron] 192.168.121.189 "GET /v2.0/subnets?id=5cce10c1-b3c3-4e3f-b709-2066dcc2c785&id=28dfb2ed-2662-4638-a2d4-9369c64b2f3c HTTP/1.1" status: 200 len: 1599 time: 0.0369835 May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-a6779a14-962e-461e-b806-be974da65e35 service neutron] 192.168.121.189 "GET /v2.0/ports?network_id=336d1bc4-69c6-4b70-b48c-36b190d979db&device_owner=network%3Adhcp HTTP/1.1" status: 200 len: 1441 time: 0.0365410 May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-7db1d18d-7e32-4a44-a9e8-e855fd16a079 service neutron] 192.168.121.189 "GET /v2.0/ports?network_id=336d1bc4-69c6-4b70-b48c-36b190d979db&device_owner=network%3Adhcp HTTP/1.1" status: 200 len: 1441 time: 0.0359824 May 21 13:29:48 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-8a3492de-0578-4c0b-bab6-4bf86f630196 service neutron] 192.168.121.189 "GET /v2.0/networks/336d1bc4-69c6-4b70-b48c-36b190d979db?fields=segments HTTP/1.1" status: 200 len: 212 time: 0.0682638 May 21 13:29:49 devstack neutron-server[3378]: INFO neutron.wsgi [req-4b5fbb4e-a3a6-45f6-b54f-383200d8dccf req-211dc48a-64e4-4e49-b630-88d45c543861 service neutron] 192.168.121.189 "GET /v2.0/networks/336d1bc4-69c6-4b70-b48c-36b190d979db?fields=provider%3Aphysical_network&fields=provider%3Anetwork_type HTTP/1.1" status: 200 len: 277 time: 0.0675299 May 21 13:29:56 devstack neutron-server[3378]: DEBUG neutron.wsgi [-] (3395) accepted ('192.168.121.189', 54086) {{(pid=3395) server /usr/local/lib/python3.6/site-packages/eventlet/wsgi.py:985}} May 21 13:29:56 devstack neutron-server[3378]: INFO neutron.api.v2.resource [None req-c451591f-e2b5-45d3-964e-e26a45f847bb demo admin] show failed (client error): The resource could not be found. May 21 13:29:56 devstack neutron-server[3378]: INFO neutron.wsgi [None req-c451591f-e2b5-45d3-964e-e26a45f847bb demo admin] 192.168.121.189 "GET /v2.0/routers/router1 HTTP/1.1" status: 404 len: 312 time: 0.0663443 May 21 13:29:57 devstack neutron-server[3378]: INFO neutron.wsgi [None req-82148186-c9ef-4e98-977c-0efbfeafbf84 demo admin] 192.168.121.189 "GET /v2.0/routers?name=router1 HTTP/1.1" status: 200 len: 859 time: 0.1023729 May 21 13:29:57 devstack neutron-server[3378]: DEBUG neutron.api.v2.base [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Request body: {'router': {'name': 'router-flat'}} {{(pid=3395) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:719}} May 21 13:29:57 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, precommit_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}} May 21 13:29:57 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, after_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Chassis candidates with external connectivity: ['30a0cb52-356d-4f2d-be32-821652201fc7'] {{(pid=3395) get_candidates_for_scheduling /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py:1412}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron.scheduler.l3_ovn_scheduler [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Gateway lrp-d7c38c52-9350-47bf-9275-118a24538a9f scheduled on chassis ['30a0cb52-356d-4f2d-be32-821652201fc7'] {{(pid=3395) _schedule_gateway /opt/stack/neutron/neutron/scheduler/l3_ovn_scheduler.py:85}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(name=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, resource={'id': '29ead671-3957-4bf2-9dbf-e1bff0a12ab2', 'name': 'router-flat', 'tenant_id': '9635d9bacb954ae78b28222e6eba4070', 'admin_state_up': True, 'status': 'ACTIVE', 'external_gateway_info': {'network_id': 'c1b6e05b-b622-4565-8a46-c03cd25b7c4b', 'external_fixed_ips': [{'subnet_id': 'ed6092d5-35d2-4dce-b347-dfeba6a5dd7f', 'ip_address': '172.24.4.51'}, {'subnet_id': 'c2e143e5-8f48-45e0-b23e-6bb497381e7b', 'ip_address': '2001:db8::1'}], 'enable_snat': True}, 'gw_port_id': 'd7c38c52-9350-47bf-9275-118a24538a9f', 'description': '', 'routes': [], 'tags': [], 'created_at': '2020-05-20T11:27:32Z', 'updated_at': '2020-05-21T13:29:57Z', 'revision_number': 9, 'project_id': '9635d9bacb954ae78b28222e6eba4070'}, resource_type=routers, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DeleteLRouterExtGwCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DeleteNATRuleInLRouterCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, type=snat, logical_ip=10.0.0.0/26, external_ip=172.24.4.51, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, ip_prefix=0.0.0.0/0, nexthop=172.24.4.1, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, ip_prefix=::/0, nexthop=2001:db8::2, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelLRouterPortCommand(name=lrp-d7c38c52-9350-47bf-9275-118a24538a9f, lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): AddLRouterPortCommand(name=lrp-d7c38c52-9350-47bf-9275-118a24538a9f, lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, may_exist=True, columns={'mac': 'fa:16:3e:54:cf:1a', 'networks': ['2001:db8::1/64', '172.24.4.51/24'], 'external_ids': {'neutron:revision_number': '6', 'neutron:subnet_ids': 'ed6092d5-35d2-4dce-b347-dfeba6a5dd7f c2e143e5-8f48-45e0-b23e-6bb497381e7b', 'neutron:network_name': 'neutron-c1b6e05b-b622-4565-8a46-c03cd25b7c4b', 'neutron:router_name': '29ead671-3957-4bf2-9dbf-e1bff0a12ab2'}, 'options': {}, 'gateway_chassis': ['30a0cb52-356d-4f2d-be32-821652201fc7']}) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): SetLRouterPortInLSwitchPortCommand(lswitch_port=d7c38c52-9350-47bf-9275-118a24538a9f, lrouter_port=lrp-d7c38c52-9350-47bf-9275-118a24538a9f, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): AddStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, columns={'ip_prefix': '0.0.0.0/0', 'nexthop': '172.24.4.1', 'external_ids': {'neutron:is_ext_gw': 'true', 'neutron:subnet_id': 'ed6092d5-35d2-4dce-b347-dfeba6a5dd7f'}}) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): AddStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, columns={'ip_prefix': '::/0', 'nexthop': '2001:db8::2', 'external_ids': {'neutron:is_ext_gw': 'true', 'neutron:subnet_id': 'c2e143e5-8f48-45e0-b23e-6bb497381e7b'}}) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): AddNATRuleInLRouterCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, columns={'type': 'snat', 'logical_ip': '10.0.0.0/26', 'external_ip': '172.24.4.51'}) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): UpdateLRouterCommand(name=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, columns={'external_ids': {'neutron:router_name': 'router-flat', 'neutron:gw_port_id': 'd7c38c52-9350-47bf-9275-118a24538a9f', 'neutron:revision_number': '9'}, 'enabled': True}, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, ip_prefix=0.0.0.0/0, nexthop=172.24.4.1, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRouteCommand(lrouter=neutron-29ead671-3957-4bf2-9dbf-e1bff0a12ab2, ip_prefix=::/0, nexthop=2001:db8::2, if_exists=True) {{(pid=3395) do_commit /usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:87}} May 21 13:29:58 devstack neutron-server[3378]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"} May 21 13:29:58 devstack neutron-server[3378]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Traceback (most recent call last): May 21 13:29:58 devstack neutron-server[3378]: File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run May 21 13:29:58 devstack neutron-server[3378]: txn.results.put(txn.do_commit()) May 21 13:29:58 devstack neutron-server[3378]: File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit May 21 13:29:58 devstack neutron-server[3378]: raise RuntimeError(msg) May 21 13:29:58 devstack neutron-server[3378]: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"} May 21 13:29:58 devstack neutron-server[3378]: May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Unable to update router 29ead671-3957-4bf2-9dbf-e1bff0a12ab2. Error: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"} May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Unable to update lrouter for 29ead671-3957-4bf2-9dbf-e1bff0a12ab2: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"} May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin Traceback (most recent call last): May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 142, in update_router May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin original_router) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1374, in update_router May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin 'error': e}) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin self.force_reraise() May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin six.reraise(self.type_, self.value, self.tb) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin raise value May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1356, in update_router May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin context, router_id, added, removed, txn=txn) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin next(self.gen) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 185, in transaction May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin yield t May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin next(self.gen) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin del self._nested_txns_map[cur_thread_id] May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin self.result = self.commit() May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin raise result.ex May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin txn.results.put(txn.do_commit()) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin raise RuntimeError(msg) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"} May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.services.ovn_l3.plugin May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron.db.extraroute_db [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Added routes are [] {{(pid=3395) _update_extra_routes /opt/stack/neutron/neutron/db/extraroute_db.py:113}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron.db.extraroute_db [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Removed routes are [] {{(pid=3395) _update_extra_routes /opt/stack/neutron/neutron/db/extraroute_db.py:121}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, precommit_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, precommit_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}} May 21 13:29:58 devstack neutron-server[3378]: DEBUG neutron_lib.callbacks.manager [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Notify callbacks [] for router, precommit_update {{(pid=3395) _notify_loop /usr/local/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193}} May 21 13:29:58 devstack neutron-server[3378]: ERROR root [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 142, in update_router\n original_router)\n', ' File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1374, in update_router\n \'error\': e})\n', ' File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1356, in update_router\n context, router_id, added, removed, txn=txn)\n', ' File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__\n next(self.gen)\n', ' File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 185, in transaction\n yield t\n', ' File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__\n next(self.gen)\n', ' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction\n del self._nested_txns_map[cur_thread_id]\n', ' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__\n self.result = self.commit()\n', ' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit\n raise result.ex\n', ' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run\n txn.results.put(txn.do_commit())\n', ' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit\n raise RuntimeError(msg)\n', 'RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"}\n']: sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <Router at 0x7ff80bfad400> is not bound to a Session; lazy load operation of attribute 'gw_port' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] update failed: No details.: sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <Router at 0x7ff80bfad400> is not bound to a Session; lazy load operation of attribute 'gw_port' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource Traceback (most recent call last): May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 142, in update_router May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource original_router) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1374, in update_router May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource 'error': e}) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise() May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1356, in update_router May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource context, router_id, added, removed, txn=txn) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource next(self.gen) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 185, in transaction May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource yield t May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource next(self.gen) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource del self._nested_txns_map[cur_thread_id] May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.result = self.commit() May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise result.ex May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource txn.results.put(txn.do_commit()) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise RuntimeError(msg) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 07e173f4-7266-4507-9ce0-d7fbe07ac65f references nonexistent row 65953028-34f9-4e4f-98ad-b0511bb566e7 in table Logical_Router_Static_Route.","error":"referential integrity violation"} May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource During handling of the above exception, another exception occurred: May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource Traceback (most recent call last): May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/resource.py", line 98, in resource May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource result = method(request=request, **args) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 625, in update May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return self._update(request, id, body, **kwargs) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise() May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return f(*args, **kwargs) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource ectxt.value = e.inner_exc May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise() May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return f(*args, **kwargs) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise() May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 685, in _update May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource obj = obj_updater(request.context, id, **kwargs) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 148, in update_router May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource revert_router) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/extraroute_db.py", line 72, in update_router May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource context, id, router) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/l3_db.py", line 1963, in update_router May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource id, router) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return method(*args, **kwargs) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise() May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return f(*args, **kwargs) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource ectxt.value = e.inner_exc May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise() May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return f(*args, **kwargs) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource self.force_reraise() May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource raise value May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/l3_db.py", line 287, in update_router May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource updated = self._make_router_dict(router_db) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/l3_db.py", line 198, in _make_router_dict May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource 'network_id': router.gw_port['network_id'], May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 287, in __get__ May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource return self.impl.get(instance_state(instance), dict_) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 723, in get May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource value = self.callable_(state, passive) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 710, in _load_for_state May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource % (orm_util.state_str(state), self.key) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <Router at 0x7ff80bfad400> is not bound to a Session; lazy load operation of attribute 'gw_port' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3) May 21 13:29:58 devstack neutron-server[3378]: ERROR neutron.api.v2.resource May 21 13:29:58 devstack neutron-server[3378]: INFO neutron.wsgi [None req-00d26082-96ea-434c-92d4-ac801feeb198 demo admin] 192.168.121.189 "PUT /v2.0/routers/29ead671-3957-4bf2-9dbf-e1bff0a12ab2 HTTP/1.1" status: 500 len: 368 time: 1.8027527 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1879950/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
