Public bug reported: In a multi-tenant setup, with a shared provider network, adding a static route to a router returns internal server error. Note: IP addresses are obfucated
openstack router set --route destination='300.300.0.0/16',gateway='172.31.96.1' tickets_private_router HttpException: 500: Server Error for url: https://controller01:9696/v2.0/routers/78c02b65-9819-48d9-9700-f045a28909e0, Request Failed: internal server error while processing your request. Openstack release: Antelope (2023.1) Neutron release: 22.0.0-1 OVN: ovn22.09 Distribution: Centos 9 Streams Manual Installation. The router: # openstack router show tickets_private_router +-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2023-06-14T17:49:52Z | | description | | | enable_ndp_proxy | None | | external_gateway_info | {"network_id": "b0c5df5c-8d5c-4fc2-babf-829421243017", "external_fixed_ips": [{"subnet_id": "4b9540d4-e3d4-403d-9918-98f111d8fedd", "ip_address": "172.31.97.226"}], "enable_snat": true} | | flavor_id | None | | id | 78c02b65-9819-48d9-9700-f045a28909e0 | | interfaces_info | [{"port_id": "7f628431-87bf-460e-88a0-b0426040a07c", "ip_address": "172.31.100.1", "subnet_id": "dd636d3d-7dc9-472e-b340-095ff85db1c2"}] | | name | tickets_private_router | | project_id | d9e5cee8a9e94a1883376ce48e676ca4 | | revision_number | 10 | | routes | | | status | ACTIVE | | tags | | | tenant_id | d9e5cee8a9e94a1883376ce48e676ca4 | | updated_at | 2023-06-16T16:58:52Z | +-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ In the logs for neutron-server: 2023-06-16 13:17:39.769 10 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-beb7e61c-a6da-4055-b41f-2bc728061394 - - - - - -] OVSDB Error: {"details":"Table Logical_Router column static_routes row 3fae901a-c2e3-468b-b8af-4343b6c3b937 references nonexistent row 57183168-288c-4700-b651-a87e3f2bf4eb in table Logical_Router_Static_Route.","error":"referential integrity violation"} neutron-server_1 | 2023-06-16 13:17:39.769 10 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-79ee6c1d-3c7b-4578-aedb-9af1dbb13e60 13d6e02fd8b74011b0e5d600d5c64300 cbadbe9f0cce4066b722422b3844a09e - - default default] Traceback (most recent call last): neutron-server_1 | File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run neutron-server_1 | txn.results.put(txn.do_commit()) neutron-server_1 | File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit neutron-server_1 | raise RuntimeError(msg) neutron-server_1 | RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 3fae901a-c2e3-468b-b8af-4343b6c3b937 references nonexistent row 57183168-288c-4700-b651-a87e3f2bf4eb in table Logical_Router_Static_Route.","error":"referential integrity violation"} neutron-server_1 | neutron-server_1 | 2023-06-16 13:17:39.769 10 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-79ee6c1d-3c7b-4578-aedb-9af1dbb13e60 13d6e02fd8b74011b0e5d600d5c64300 cbadbe9f0cce4066b722422b3844a09e - - default default] Unable to update router 78c02b65-9819-48d9-9700-f045a28909e0. Error: OVSDB Error: {"details":"Table Logical_Router column static_routes row 3fae901a-c2e3-468b-b8af-4343b6c3b937 references nonexistent row 57183168-288c-4700-b651-a87e3f2bf4eb in table Logical_Router_Static_Route.","error":"referential integrity violation"}: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 3fae901a-c2e3-468b-b8af-4343b6c3b937 references nonexistent row 57183168-288c-4700-b651-a87e3f2bf4eb in table Logical_Router_Static_Route.","error":"referential integrity violation"} neutron-server_1 | 2023-06-16 13:17:39.770 10 ERROR neutron.services.ovn_l3.plugin [None req-79ee6c1d-3c7b-4578-aedb-9af1dbb13e60 13d6e02fd8b74011b0e5d600d5c64300 cbadbe9f0cce4066b722422b3844a09e - - default default] Unable to update lrouter for 78c02b65-9819-48d9-9700-f045a28909e0: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 3fae901a-c2e3-468b-b8af-4343b6c3b937 references nonexistent row 57183168-288c-4700-b651-a87e3f2bf4eb in table Logical_Router_Static_Route.","error":"referential integrity violation"} Stack trace follows. This will ocassionaly succeed on other routers after a delay of 10+ minutes, but some routers refuse to accept the staic route. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2024251 Title: Adding a static route to router returns internal server error Status in neutron: New Bug description: In a multi-tenant setup, with a shared provider network, adding a static route to a router returns internal server error. Note: IP addresses are obfucated openstack router set --route destination='300.300.0.0/16',gateway='172.31.96.1' tickets_private_router HttpException: 500: Server Error for url: https://controller01:9696/v2.0/routers/78c02b65-9819-48d9-9700-f045a28909e0, Request Failed: internal server error while processing your request. Openstack release: Antelope (2023.1) Neutron release: 22.0.0-1 OVN: ovn22.09 Distribution: Centos 9 Streams Manual Installation. The router: # openstack router show tickets_private_router +-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2023-06-14T17:49:52Z | | description | | | enable_ndp_proxy | None | | external_gateway_info | {"network_id": "b0c5df5c-8d5c-4fc2-babf-829421243017", "external_fixed_ips": [{"subnet_id": "4b9540d4-e3d4-403d-9918-98f111d8fedd", "ip_address": "172.31.97.226"}], "enable_snat": true} | | flavor_id | None | | id | 78c02b65-9819-48d9-9700-f045a28909e0 | | interfaces_info | [{"port_id": "7f628431-87bf-460e-88a0-b0426040a07c", "ip_address": "172.31.100.1", "subnet_id": "dd636d3d-7dc9-472e-b340-095ff85db1c2"}] | | name | tickets_private_router | | project_id | d9e5cee8a9e94a1883376ce48e676ca4 | | revision_number | 10 | | routes | | | status | ACTIVE | | tags | | | tenant_id | d9e5cee8a9e94a1883376ce48e676ca4 | | updated_at | 2023-06-16T16:58:52Z | +-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ In the logs for neutron-server: 2023-06-16 13:17:39.769 10 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-beb7e61c-a6da-4055-b41f-2bc728061394 - - - - - -] OVSDB Error: {"details":"Table Logical_Router column static_routes row 3fae901a-c2e3-468b-b8af-4343b6c3b937 references nonexistent row 57183168-288c-4700-b651-a87e3f2bf4eb in table Logical_Router_Static_Route.","error":"referential integrity violation"} neutron-server_1 | 2023-06-16 13:17:39.769 10 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-79ee6c1d-3c7b-4578-aedb-9af1dbb13e60 13d6e02fd8b74011b0e5d600d5c64300 cbadbe9f0cce4066b722422b3844a09e - - default default] Traceback (most recent call last): neutron-server_1 | File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run neutron-server_1 | txn.results.put(txn.do_commit()) neutron-server_1 | File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit neutron-server_1 | raise RuntimeError(msg) neutron-server_1 | RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 3fae901a-c2e3-468b-b8af-4343b6c3b937 references nonexistent row 57183168-288c-4700-b651-a87e3f2bf4eb in table Logical_Router_Static_Route.","error":"referential integrity violation"} neutron-server_1 | neutron-server_1 | 2023-06-16 13:17:39.769 10 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-79ee6c1d-3c7b-4578-aedb-9af1dbb13e60 13d6e02fd8b74011b0e5d600d5c64300 cbadbe9f0cce4066b722422b3844a09e - - default default] Unable to update router 78c02b65-9819-48d9-9700-f045a28909e0. Error: OVSDB Error: {"details":"Table Logical_Router column static_routes row 3fae901a-c2e3-468b-b8af-4343b6c3b937 references nonexistent row 57183168-288c-4700-b651-a87e3f2bf4eb in table Logical_Router_Static_Route.","error":"referential integrity violation"}: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 3fae901a-c2e3-468b-b8af-4343b6c3b937 references nonexistent row 57183168-288c-4700-b651-a87e3f2bf4eb in table Logical_Router_Static_Route.","error":"referential integrity violation"} neutron-server_1 | 2023-06-16 13:17:39.770 10 ERROR neutron.services.ovn_l3.plugin [None req-79ee6c1d-3c7b-4578-aedb-9af1dbb13e60 13d6e02fd8b74011b0e5d600d5c64300 cbadbe9f0cce4066b722422b3844a09e - - default default] Unable to update lrouter for 78c02b65-9819-48d9-9700-f045a28909e0: RuntimeError: OVSDB Error: {"details":"Table Logical_Router column static_routes row 3fae901a-c2e3-468b-b8af-4343b6c3b937 references nonexistent row 57183168-288c-4700-b651-a87e3f2bf4eb in table Logical_Router_Static_Route.","error":"referential integrity violation"} Stack trace follows. This will ocassionaly succeed on other routers after a delay of 10+ minutes, but some routers refuse to accept the staic route. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2024251/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
