Public bug reported:
Under some conditions, Neutron tries to create a record in the Database
a few times and fails with an unexpected exception that should be
handled by existing code.
Neutron mode: ovs+dvr
How to reproduce:
1. Tenant network connected to distributed router
2. VM with tenant network created or migrated to an empty hypervisor
When VM is placed to empty node L3-agent notifies neutron-server to bind router
distributed port to node and we get ERROR traceback in neutron-server log:
```
ERROR oslo_messaging.rpc.server [parameters: {'port_id':
'f269cf3d-8a00-4b5b-99c2-63cae0968f42', 'host': 'gm-os-hv-017', 'router_id':
'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3', 'vif_type': 'unbound', 'vif_details':
'', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}]
ERROR oslo_messaging.rpc.server [SQL: INSERT INTO ml2_distributed_port_bindings
(port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status)
VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s,
%(vnic_type)s, %(profile)s, %(status)s)]
ERROR oslo_messaging.rpc.server sqlalchemy.exc.PendingRollbackError: This
Session's transaction has been rolled back due to a previous exception during
flush. To begin a new transaction with this Session, first issue
Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062,
"Duplicate entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key
'PRIMARY'")
ERROR oslo_messaging.rpc.server raise sa_exc.PendingRollbackError(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in
_assert_active
ERROR oslo_messaging.rpc.server self._assert_active()
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in
_connection_for_bind
ERROR oslo_messaging.rpc.server return
self._transaction._connection_for_bind(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in
_connection_for_bind
ERROR oslo_messaging.rpc.server conn = self._connection_for_bind(bind,
close_with_result=True)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in
execute
ERROR oslo_messaging.rpc.server result = self.session.execute(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in
_iter
ERROR oslo_messaging.rpc.server return self._iter().one()
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one
ERROR oslo_messaging.rpc.server return
(context.session.query(models.DistributedPortBinding).
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in
ensure_distributed_port_binding
ERROR oslo_messaging.rpc.server binding =
db.ensure_distributed_port_binding(
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in
update_distributed_port_binding
ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped
ERROR oslo_messaging.rpc.server raise self.value
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable
exception: %s", e)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped
ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper
ERROR oslo_messaging.rpc.server raise self.value
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper
ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped
ERROR oslo_messaging.rpc.server raise self.value
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped
ERROR oslo_messaging.rpc.server return method(*args, **kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped
ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner
ERROR oslo_messaging.rpc.server self.plugin.update_distributed_port_binding(
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line
253, in _ensure_host_set_on_port
ERROR oslo_messaging.rpc.server self._ensure_host_set_on_port(
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line
178, in _ensure_host_set_on_ports
ERROR oslo_messaging.rpc.server self._ensure_host_set_on_ports(context,
host, routers)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line
131, in sync_routers
ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped
ERROR oslo_messaging.rpc.server raise self.value
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable
exception: %s", e)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped
ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper
ERROR oslo_messaging.rpc.server raise self.value
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper
ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped
ERROR oslo_messaging.rpc.server raise self.value
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped
ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229,
in _do_dispatch
ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method,
ctxt, args)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309,
in dispatch
ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in
_process_incoming
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server
ERROR oslo_messaging.rpc.server During handling of the above exception, another
exception occurred:
ERROR oslo_messaging.rpc.server
ERROR oslo_messaging.rpc.server (Background on this error at:
https://sqlalche.me/e/14/gkpj)
ERROR oslo_messaging.rpc.server [parameters: {'port_id':
'f269cf3d-8a00-4b5b-99c2-63cae0968f42', 'host': 'gm-os-hv-017', 'router_id':
'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3', 'vif_type': 'unbound', 'vif_details':
'', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}]
ERROR oslo_messaging.rpc.server [SQL: INSERT INTO ml2_distributed_port_bindings
(port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status)
VALUES (%(port_id)s, %(host)s, %(router_id)s, %(vif_type)s, %(vif_details)s,
%(vnic_type)s, %(profile)s, %(status)s)]
ERROR oslo_messaging.rpc.server oslo_db.exception.DBDuplicateEntry:
(pymysql.err.IntegrityError) (1062, "Duplicate entry
'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key 'PRIMARY'")
ERROR oslo_messaging.rpc.server raise errorclass(errno, errval)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in
raise_mysql_exception
ERROR oslo_messaging.rpc.server err.raise_mysql_exception(self._data)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in
raise_for_error
ERROR oslo_messaging.rpc.server packet.raise_for_error()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in
_read_packet
ERROR oslo_messaging.rpc.server first_packet =
self.connection._read_packet()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
ERROR oslo_messaging.rpc.server result.read()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in
_read_query_result
ERROR oslo_messaging.rpc.server self._affected_rows =
self._read_query_result(unbuffered=unbuffered)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
ERROR oslo_messaging.rpc.server conn.query(q)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
ERROR oslo_messaging.rpc.server result = self._query(query)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
ERROR oslo_messaging.rpc.server cursor.execute(statement, parameters)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in
do_execute
ERROR oslo_messaging.rpc.server self.dialect.do_execute(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in
_execute_context
ERROR oslo_messaging.rpc.server raise exception
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in
raise_
ERROR oslo_messaging.rpc.server util.raise_(newraise,
with_traceback=exc_info[2], from_=e)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in
_handle_dbapi_exception
ERROR oslo_messaging.rpc.server self._handle_dbapi_exception(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in
_execute_context
ERROR oslo_messaging.rpc.server ret = self._execute_context(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in
_execute_clauseelement
ERROR oslo_messaging.rpc.server return connection._execute_clauseelement(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in
_execute_on_connection
ERROR oslo_messaging.rpc.server return meth(self, args_10style,
kwargs_10style, execution_options)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in
_execute_20
ERROR oslo_messaging.rpc.server c = connection._execute_20(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097,
in _emit_insert_statements
ERROR oslo_messaging.rpc.server _emit_insert_statements(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245,
in save_obj
ERROR oslo_messaging.rpc.server util.preloaded.orm_persistence.save_obj(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in
execute
ERROR oslo_messaging.rpc.server rec.execute(self)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in
execute
ERROR oslo_messaging.rpc.server flush_context.execute()
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in
_flush
ERROR oslo_messaging.rpc.server raise exception
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in
raise_
ERROR oslo_messaging.rpc.server compat.raise_(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70,
in __exit__
ERROR oslo_messaging.rpc.server
transaction.rollback(_capture_exception=True)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in
_flush
ERROR oslo_messaging.rpc.server self._flush(objects)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in
flush
ERROR oslo_messaging.rpc.server self.session.flush()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line
676, in _session
ERROR oslo_messaging.rpc.server next(self.gen)
ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.9/contextlib.py",
line 126, in __exit__
ERROR oslo_messaging.rpc.server yield resource
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line
1060, in _transaction_scope
ERROR oslo_messaging.rpc.server next(self.gen)
ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.9/contextlib.py",
line 126, in __exit__
ERROR oslo_messaging.rpc.server return record
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in
ensure_distributed_port_binding
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server
ERROR oslo_messaging.rpc.server The above exception was the direct cause of the
following exception:
ERROR oslo_messaging.rpc.server
ERROR oslo_messaging.rpc.server pymysql.err.IntegrityError: (1062, "Duplicate
entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key 'PRIMARY'")
ERROR oslo_messaging.rpc.server raise errorclass(errno, errval)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in
raise_mysql_exception
ERROR oslo_messaging.rpc.server err.raise_mysql_exception(self._data)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in
raise_for_error
ERROR oslo_messaging.rpc.server packet.raise_for_error()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in
_read_packet
ERROR oslo_messaging.rpc.server first_packet =
self.connection._read_packet()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
ERROR oslo_messaging.rpc.server result.read()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in
_read_query_result
ERROR oslo_messaging.rpc.server self._affected_rows =
self._read_query_result(unbuffered=unbuffered)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
ERROR oslo_messaging.rpc.server conn.query(q)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
ERROR oslo_messaging.rpc.server result = self._query(query)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
ERROR oslo_messaging.rpc.server cursor.execute(statement, parameters)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in
do_execute
ERROR oslo_messaging.rpc.server self.dialect.do_execute(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in
_execute_context
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server [req-1112fb3c-05bc-4754-b4d9-e0813bba902c - - -
- -] Exception during message handling: sqlalchemy.exc.PendingRollbackError:
This Session's transaction has been rolled back due to a previous exception
during flush. To begin a new transaction with this Session, first issue
Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062,
"Duplicate entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key
'PRIMARY'")
```
As you can see neutron-server should handle exception for `DuplicateEntry` and
return existing record:
https://github.com/openstack/neutron/blob/20.3.1/neutron/plugins/ml2/db.py#L111
But this code is not doing it and failed with:
```
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled
back due to a previous exception during flush. To begin a new transaction with
this Session, first issue Session.rollback(). Original exception was:
(pymysql.err.IntegrityError) (1062, "Duplicate entry ..."
```
It seems that this part of the code is not covered by tests and not work
correctly.
It is reproduced on our side only in one production system and we can
not reproduce it on our lab env.
Environment
===========
AlmaLinux 9.5
```
openstack-neutron-20.3.1-1.el9s
python-oslo-db-11.2.0-1.el9s
python-oslo-messaging-12.13.3-1.el9s
python-sqlalchemy-1.4.45-3.el9
```
** Affects: neutron
Importance: Undecided
Status: New
** Description changed:
Under some conditions, Neutron tries to create a record in the Database
a few times and fails with an unexpected exception that should be
handled by existing code.
Neutron mode: ovs+dvr
How to reproduce:
1. Tenant network connected to distributed router
2. VM with tenant network created or migrated to an empty hypervisor
When VM is placed to empty node L3-agent notifies neutron-server to bind
router distributed port to node and we get ERROR traceback in neutron-server
log:
```
ERROR oslo_messaging.rpc.server [parameters: {'port_id':
'f269cf3d-8a00-4b5b-99c2-63cae0968f42', 'host': 'gm-os-hv-017', 'router_id':
'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3', 'vif_type': 'unbound', 'vif_details':
'', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}]
ERROR oslo_messaging.rpc.server [SQL: INSERT INTO
ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details,
vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s,
%(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]
ERROR oslo_messaging.rpc.server sqlalchemy.exc.PendingRollbackError: This
Session's transaction has been rolled back due to a previous exception during
flush. To begin a new transaction with this Session, first issue
Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062,
"Duplicate entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key
'PRIMARY'")
ERROR oslo_messaging.rpc.server raise sa_exc.PendingRollbackError(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in
_assert_active
ERROR oslo_messaging.rpc.server self._assert_active()
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in
_connection_for_bind
ERROR oslo_messaging.rpc.server return
self._transaction._connection_for_bind(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in
_connection_for_bind
ERROR oslo_messaging.rpc.server conn = self._connection_for_bind(bind,
close_with_result=True)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in
execute
ERROR oslo_messaging.rpc.server result = self.session.execute(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in
_iter
ERROR oslo_messaging.rpc.server return self._iter().one()
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one
ERROR oslo_messaging.rpc.server return
(context.session.query(models.DistributedPortBinding).
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in
ensure_distributed_port_binding
ERROR oslo_messaging.rpc.server binding =
db.ensure_distributed_port_binding(
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in
update_distributed_port_binding
ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped
ERROR oslo_messaging.rpc.server raise self.value
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable
exception: %s", e)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped
ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper
ERROR oslo_messaging.rpc.server raise self.value
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper
ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped
ERROR oslo_messaging.rpc.server raise self.value
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped
ERROR oslo_messaging.rpc.server return method(*args, **kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped
ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner
ERROR oslo_messaging.rpc.server
self.plugin.update_distributed_port_binding(
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line
253, in _ensure_host_set_on_port
ERROR oslo_messaging.rpc.server self._ensure_host_set_on_port(
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line
178, in _ensure_host_set_on_ports
ERROR oslo_messaging.rpc.server self._ensure_host_set_on_ports(context,
host, routers)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line
131, in sync_routers
ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped
ERROR oslo_messaging.rpc.server raise self.value
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable
exception: %s", e)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped
ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper
ERROR oslo_messaging.rpc.server raise self.value
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper
ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped
ERROR oslo_messaging.rpc.server raise self.value
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped
ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229,
in _do_dispatch
ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint,
method, ctxt, args)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309,
in dispatch
ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in
_process_incoming
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server
ERROR oslo_messaging.rpc.server During handling of the above exception,
another exception occurred:
ERROR oslo_messaging.rpc.server
ERROR oslo_messaging.rpc.server (Background on this error at:
https://sqlalche.me/e/14/gkpj)
ERROR oslo_messaging.rpc.server [parameters: {'port_id':
'f269cf3d-8a00-4b5b-99c2-63cae0968f42', 'host': 'gm-os-hv-017', 'router_id':
'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3', 'vif_type': 'unbound', 'vif_details':
'', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}]
ERROR oslo_messaging.rpc.server [SQL: INSERT INTO
ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details,
vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s,
%(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]
ERROR oslo_messaging.rpc.server oslo_db.exception.DBDuplicateEntry:
(pymysql.err.IntegrityError) (1062, "Duplicate entry
'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key 'PRIMARY'")
ERROR oslo_messaging.rpc.server raise errorclass(errno, errval)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in
raise_mysql_exception
ERROR oslo_messaging.rpc.server err.raise_mysql_exception(self._data)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in
raise_for_error
ERROR oslo_messaging.rpc.server packet.raise_for_error()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in
_read_packet
ERROR oslo_messaging.rpc.server first_packet =
self.connection._read_packet()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
ERROR oslo_messaging.rpc.server result.read()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in
_read_query_result
ERROR oslo_messaging.rpc.server self._affected_rows =
self._read_query_result(unbuffered=unbuffered)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
ERROR oslo_messaging.rpc.server conn.query(q)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
ERROR oslo_messaging.rpc.server result = self._query(query)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
ERROR oslo_messaging.rpc.server cursor.execute(statement, parameters)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in
do_execute
ERROR oslo_messaging.rpc.server self.dialect.do_execute(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in
_execute_context
ERROR oslo_messaging.rpc.server raise exception
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in
raise_
ERROR oslo_messaging.rpc.server util.raise_(newraise,
with_traceback=exc_info[2], from_=e)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in
_handle_dbapi_exception
ERROR oslo_messaging.rpc.server self._handle_dbapi_exception(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in
_execute_context
ERROR oslo_messaging.rpc.server ret = self._execute_context(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in
_execute_clauseelement
ERROR oslo_messaging.rpc.server return connection._execute_clauseelement(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in
_execute_on_connection
ERROR oslo_messaging.rpc.server return meth(self, args_10style,
kwargs_10style, execution_options)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in
_execute_20
ERROR oslo_messaging.rpc.server c = connection._execute_20(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097,
in _emit_insert_statements
ERROR oslo_messaging.rpc.server _emit_insert_statements(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245,
in save_obj
ERROR oslo_messaging.rpc.server util.preloaded.orm_persistence.save_obj(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in
execute
ERROR oslo_messaging.rpc.server rec.execute(self)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in
execute
ERROR oslo_messaging.rpc.server flush_context.execute()
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in
_flush
ERROR oslo_messaging.rpc.server raise exception
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in
raise_
ERROR oslo_messaging.rpc.server compat.raise_(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70,
in __exit__
ERROR oslo_messaging.rpc.server
transaction.rollback(_capture_exception=True)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in
_flush
ERROR oslo_messaging.rpc.server self._flush(objects)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in
flush
ERROR oslo_messaging.rpc.server self.session.flush()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line
676, in _session
ERROR oslo_messaging.rpc.server next(self.gen)
ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.9/contextlib.py",
line 126, in __exit__
ERROR oslo_messaging.rpc.server yield resource
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line
1060, in _transaction_scope
ERROR oslo_messaging.rpc.server next(self.gen)
ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.9/contextlib.py",
line 126, in __exit__
ERROR oslo_messaging.rpc.server return record
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in
ensure_distributed_port_binding
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server
ERROR oslo_messaging.rpc.server The above exception was the direct cause of
the following exception:
ERROR oslo_messaging.rpc.server
ERROR oslo_messaging.rpc.server pymysql.err.IntegrityError: (1062, "Duplicate
entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key 'PRIMARY'")
ERROR oslo_messaging.rpc.server raise errorclass(errno, errval)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in
raise_mysql_exception
ERROR oslo_messaging.rpc.server err.raise_mysql_exception(self._data)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in
raise_for_error
ERROR oslo_messaging.rpc.server packet.raise_for_error()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in
_read_packet
ERROR oslo_messaging.rpc.server first_packet =
self.connection._read_packet()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
ERROR oslo_messaging.rpc.server result.read()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in
_read_query_result
ERROR oslo_messaging.rpc.server self._affected_rows =
self._read_query_result(unbuffered=unbuffered)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
ERROR oslo_messaging.rpc.server conn.query(q)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
ERROR oslo_messaging.rpc.server result = self._query(query)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
ERROR oslo_messaging.rpc.server cursor.execute(statement, parameters)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in
do_execute
ERROR oslo_messaging.rpc.server self.dialect.do_execute(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in
_execute_context
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server [req-1112fb3c-05bc-4754-b4d9-e0813bba902c - -
- - -] Exception during message handling: sqlalchemy.exc.PendingRollbackError:
This Session's transaction has been rolled back due to a previous exception
during flush. To begin a new transaction with this Session, first issue
Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062,
"Duplicate entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key
'PRIMARY'")
```
As you can see neutron-server should handle exception for `DuplicateEntry`
and return existing record:
https://github.com/openstack/neutron/blob/20.3.1/neutron/plugins/ml2/db.py#L111
But this code is not doing it and failed with:
```
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been
rolled back due to a previous exception during flush. To begin a new
transaction with this Session, first issue Session.rollback(). Original
exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry ..."
```
It seems that this part of the code is not covered by tests and not work
correctly.
It is reproduced on our side only in one production system and we can
not reproduce it on our lab env.
- versions:
+ Environment
+ ===========
+ AlmaLinux 9.5
```
openstack-neutron-20.3.1-1.el9s
python-oslo-db-11.2.0-1.el9s
python-oslo-messaging-12.13.3-1.el9s
python-sqlalchemy-1.4.45-3.el9
```
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2106374
Title:
neutron-server error with ml2_distributed_port_bindings
Status in neutron:
New
Bug description:
Under some conditions, Neutron tries to create a record in the
Database a few times and fails with an unexpected exception that
should be handled by existing code.
Neutron mode: ovs+dvr
How to reproduce:
1. Tenant network connected to distributed router
2. VM with tenant network created or migrated to an empty hypervisor
When VM is placed to empty node L3-agent notifies neutron-server to bind
router distributed port to node and we get ERROR traceback in neutron-server
log:
```
ERROR oslo_messaging.rpc.server [parameters: {'port_id':
'f269cf3d-8a00-4b5b-99c2-63cae0968f42', 'host': 'gm-os-hv-017', 'router_id':
'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3', 'vif_type': 'unbound', 'vif_details':
'', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}]
ERROR oslo_messaging.rpc.server [SQL: INSERT INTO
ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details,
vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s,
%(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]
ERROR oslo_messaging.rpc.server sqlalchemy.exc.PendingRollbackError: This
Session's transaction has been rolled back due to a previous exception during
flush. To begin a new transaction with this Session, first issue
Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062,
"Duplicate entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key
'PRIMARY'")
ERROR oslo_messaging.rpc.server raise sa_exc.PendingRollbackError(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 601, in
_assert_active
ERROR oslo_messaging.rpc.server self._assert_active()
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 721, in
_connection_for_bind
ERROR oslo_messaging.rpc.server return
self._transaction._connection_for_bind(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1552, in
_connection_for_bind
ERROR oslo_messaging.rpc.server conn = self._connection_for_bind(bind,
close_with_result=True)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1708, in
execute
ERROR oslo_messaging.rpc.server result = self.session.execute(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in
_iter
ERROR oslo_messaging.rpc.server return self._iter().one()
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 2870, in one
ERROR oslo_messaging.rpc.server return
(context.session.query(models.DistributedPortBinding).
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 114, in
ensure_distributed_port_binding
ERROR oslo_messaging.rpc.server binding =
db.ensure_distributed_port_binding(
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 2010, in
update_distributed_port_binding
ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped
ERROR oslo_messaging.rpc.server raise self.value
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable
exception: %s", e)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped
ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper
ERROR oslo_messaging.rpc.server raise self.value
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper
ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped
ERROR oslo_messaging.rpc.server raise self.value
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped
ERROR oslo_messaging.rpc.server return method(*args, **kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 218, in wrapped
ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/common/utils.py", line 701, in inner
ERROR oslo_messaging.rpc.server
self.plugin.update_distributed_port_binding(
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line
253, in _ensure_host_set_on_port
ERROR oslo_messaging.rpc.server self._ensure_host_set_on_port(
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line
178, in _ensure_host_set_on_ports
ERROR oslo_messaging.rpc.server self._ensure_host_set_on_ports(context,
host, routers)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line
131, in sync_routers
ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 179, in wrapped
ERROR oslo_messaging.rpc.server raise self.value
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable
exception: %s", e)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 183, in wrapped
ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper
ERROR oslo_messaging.rpc.server raise self.value
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper
ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped
ERROR oslo_messaging.rpc.server raise self.value
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in
force_reraise
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped
ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229,
in _do_dispatch
ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint,
method, ctxt, args)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309,
in dispatch
ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in
_process_incoming
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server
ERROR oslo_messaging.rpc.server During handling of the above exception,
another exception occurred:
ERROR oslo_messaging.rpc.server
ERROR oslo_messaging.rpc.server (Background on this error at:
https://sqlalche.me/e/14/gkpj)
ERROR oslo_messaging.rpc.server [parameters: {'port_id':
'f269cf3d-8a00-4b5b-99c2-63cae0968f42', 'host': 'gm-os-hv-017', 'router_id':
'e3fb6ba3-7cd5-40d4-bb24-4544e1319ea3', 'vif_type': 'unbound', 'vif_details':
'', 'vnic_type': 'normal', 'profile': '', 'status': 'DOWN'}]
ERROR oslo_messaging.rpc.server [SQL: INSERT INTO
ml2_distributed_port_bindings (port_id, host, router_id, vif_type, vif_details,
vnic_type, profile, status) VALUES (%(port_id)s, %(host)s, %(router_id)s,
%(vif_type)s, %(vif_details)s, %(vnic_type)s, %(profile)s, %(status)s)]
ERROR oslo_messaging.rpc.server oslo_db.exception.DBDuplicateEntry:
(pymysql.err.IntegrityError) (1062, "Duplicate entry
'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key 'PRIMARY'")
ERROR oslo_messaging.rpc.server raise errorclass(errno, errval)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in
raise_mysql_exception
ERROR oslo_messaging.rpc.server err.raise_mysql_exception(self._data)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in
raise_for_error
ERROR oslo_messaging.rpc.server packet.raise_for_error()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in
_read_packet
ERROR oslo_messaging.rpc.server first_packet =
self.connection._read_packet()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
ERROR oslo_messaging.rpc.server result.read()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in
_read_query_result
ERROR oslo_messaging.rpc.server self._affected_rows =
self._read_query_result(unbuffered=unbuffered)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
ERROR oslo_messaging.rpc.server conn.query(q)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
ERROR oslo_messaging.rpc.server result = self._query(query)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
ERROR oslo_messaging.rpc.server cursor.execute(statement, parameters)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in
do_execute
ERROR oslo_messaging.rpc.server self.dialect.do_execute(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in
_execute_context
ERROR oslo_messaging.rpc.server raise exception
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in
raise_
ERROR oslo_messaging.rpc.server util.raise_(newraise,
with_traceback=exc_info[2], from_=e)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2122, in
_handle_dbapi_exception
ERROR oslo_messaging.rpc.server self._handle_dbapi_exception(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in
_execute_context
ERROR oslo_messaging.rpc.server ret = self._execute_context(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in
_execute_clauseelement
ERROR oslo_messaging.rpc.server return connection._execute_clauseelement(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in
_execute_on_connection
ERROR oslo_messaging.rpc.server return meth(self, args_10style,
kwargs_10style, execution_options)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in
_execute_20
ERROR oslo_messaging.rpc.server c = connection._execute_20(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097,
in _emit_insert_statements
ERROR oslo_messaging.rpc.server _emit_insert_statements(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245,
in save_obj
ERROR oslo_messaging.rpc.server util.preloaded.orm_persistence.save_obj(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in
execute
ERROR oslo_messaging.rpc.server rec.execute(self)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in
execute
ERROR oslo_messaging.rpc.server flush_context.execute()
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3544, in
_flush
ERROR oslo_messaging.rpc.server raise exception
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in
raise_
ERROR oslo_messaging.rpc.server compat.raise_(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70,
in __exit__
ERROR oslo_messaging.rpc.server
transaction.rollback(_capture_exception=True)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3584, in
_flush
ERROR oslo_messaging.rpc.server self._flush(objects)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3444, in
flush
ERROR oslo_messaging.rpc.server self.session.flush()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line
676, in _session
ERROR oslo_messaging.rpc.server next(self.gen)
ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.9/contextlib.py",
line 126, in __exit__
ERROR oslo_messaging.rpc.server yield resource
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line
1060, in _transaction_scope
ERROR oslo_messaging.rpc.server next(self.gen)
ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.9/contextlib.py",
line 126, in __exit__
ERROR oslo_messaging.rpc.server return record
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/neutron/plugins/ml2/db.py", line 110, in
ensure_distributed_port_binding
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server
ERROR oslo_messaging.rpc.server The above exception was the direct cause of
the following exception:
ERROR oslo_messaging.rpc.server
ERROR oslo_messaging.rpc.server pymysql.err.IntegrityError: (1062, "Duplicate
entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key 'PRIMARY'")
ERROR oslo_messaging.rpc.server raise errorclass(errno, errval)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/err.py", line 107, in
raise_mysql_exception
ERROR oslo_messaging.rpc.server err.raise_mysql_exception(self._data)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/protocol.py", line 223, in
raise_for_error
ERROR oslo_messaging.rpc.server packet.raise_for_error()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 676, in
_read_packet
ERROR oslo_messaging.rpc.server first_packet =
self.connection._read_packet()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 1069, in read
ERROR oslo_messaging.rpc.server result.read()
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 724, in
_read_query_result
ERROR oslo_messaging.rpc.server self._affected_rows =
self._read_query_result(unbuffered=unbuffered)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/connections.py", line 505, in query
ERROR oslo_messaging.rpc.server conn.query(q)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 321, in _query
ERROR oslo_messaging.rpc.server result = self._query(query)
ERROR oslo_messaging.rpc.server File
"/usr/lib/python3.9/site-packages/pymysql/cursors.py", line 163, in execute
ERROR oslo_messaging.rpc.server cursor.execute(statement, parameters)
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in
do_execute
ERROR oslo_messaging.rpc.server self.dialect.do_execute(
ERROR oslo_messaging.rpc.server File
"/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in
_execute_context
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server [req-1112fb3c-05bc-4754-b4d9-e0813bba902c - -
- - -] Exception during message handling: sqlalchemy.exc.PendingRollbackError:
This Session's transaction has been rolled back due to a previous exception
during flush. To begin a new transaction with this Session, first issue
Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062,
"Duplicate entry 'f269cf3d-8a00-4b5b-99c2-63cae0968f42-gm-os-hv-017' for key
'PRIMARY'")
```
As you can see neutron-server should handle exception for `DuplicateEntry`
and return existing record:
https://github.com/openstack/neutron/blob/20.3.1/neutron/plugins/ml2/db.py#L111
But this code is not doing it and failed with:
```
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been
rolled back due to a previous exception during flush. To begin a new
transaction with this Session, first issue Session.rollback(). Original
exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry ..."
```
It seems that this part of the code is not covered by tests and not
work correctly.
It is reproduced on our side only in one production system and we can
not reproduce it on our lab env.
Environment
===========
AlmaLinux 9.5
```
openstack-neutron-20.3.1-1.el9s
python-oslo-db-11.2.0-1.el9s
python-oslo-messaging-12.13.3-1.el9s
python-sqlalchemy-1.4.45-3.el9
```
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2106374/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
