Reviewed: https://review.openstack.org/341795 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=890c30fc52eb1b16cb7f671206b1a0edeb7fb58f Submitter: Jenkins Branch: master
commit 890c30fc52eb1b16cb7f671206b1a0edeb7fb58f Author: Henry Gessau <ges...@gmail.com> Date: Wed Jul 13 16:14:23 2016 -0400 Add retry decorator to dhcp_ready_on_ports Enable DB retries for this provisioning blocks phase. Closes-Bug: #1602833 Change-Id: Ie6444d6552b4c1e73c00d5624e3866417204c9e8 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1602833 Title: Missing retry decorator on dhcp_ready_on_ports Status in neutron: Fix Released Bug description: 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server [req-30a8fac4-91ee-45e1-a259-19410b3a86f1 - - - - -] Exception during message handling 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 239, in dhcp_ready_on_ports 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server provisioning_blocks.DHCP_ENTITY) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/neutron/db/provisioning_blocks.py", line 141, in provisioning_complete 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server standard_attr_id): 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/neutron/db/provisioning_blocks.py", line 114, in remove_provisioning_component 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server return False 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 490, in __exit__ 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server self.rollback() 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__ 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server compat.reraise(exc_type, exc_value, exc_tb) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 487, in __exit__ 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server self.commit() 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 392, in commit 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server self._prepare_impl() 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server self.session.flush() 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2019, in flush 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server self._flush(objects) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2137, in _flush 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server transaction.rollback(_capture_exception=True) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__ 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server compat.reraise(exc_type, exc_value, exc_tb) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2101, in _flush 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server flush_context.execute() 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server rec.execute(self) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 561, in execute 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server uow 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 249, in delete_obj 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server cached_connections, mapper, table, delete) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 892, in _emit_delete_statements 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server c = connection.execute(statement, del_objects) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server return meth(self, multiparams, params) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server return connection._execute_clauseelement(self, multiparams, params) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server compiled_sql, distilled_params 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server context) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server util.raise_from_cause(newraise, exc_info) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server reraise(type(exception), exception, tb=exc_tb, cause=cause) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server context) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server cursor.execute(statement, parameters) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/pymysql/cursors.py", line 161, in execute 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server result = self._query(query) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/pymysql/cursors.py", line 317, in _query 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server conn.query(q) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/pymysql/connections.py", line 837, in query 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/pymysql/connections.py", line 1021, in _read_query_result 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server result.read() 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/pymysql/connections.py", line 1304, in read 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server first_packet = self.connection._read_packet() 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/pymysql/connections.py", line 983, in _read_packet 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server packet.check_error() 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/pymysql/connections.py", line 395, in check_error 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server err.raise_mysql_exception(self._data) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server _check_mysql_exception(errinfo) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/pymysql/err.py", line 115, in _check_mysql_exception 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server raise InternalError(errno, errorvalue) 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server DBDeadlock: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'DELETE FROM provisioningblocks WHERE provisioningblocks.standard_attr_id = %(standard_attr_id)s AND provisioningblocks.entity = %(entity)s'] [parameters: {'standard_attr_id': 14843, 'entity': u'DHCP'}] 2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1602833/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp