Public bug reported: Description =========== While run "nova list" to get instance list, NOVA report "[root@ARCHS22K8144 httpd]# nova list ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-e08c4a96-d986-4250-8186-7d5974b0a1b5)" randomly.
Steps to reproduce ================== for i in `seq 0 100`; do echo ===========================$i; nova list; sleep 3; done Expected result =============== it should list all instances successfully. Actual result ============= most time it can list instances, but some time it report "Unexpected API Error". Environment =========== 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ # rpm -qa | grep nova openstack-nova-common-14.0.3-1.el7.noarch openstack-nova-console-14.0.3-1.el7.noarch python2-novaclient-6.0.0-1.el7.noarch openstack-nova-novncproxy-14.0.3-1.el7.noarch python-nova-14.0.3-1.el7.noarch openstack-nova-api-14.0.3-1.el7.noarch openstack-nova-conductor-14.0.3-1.el7.noarch openstack-nova-scheduler-14.0.3-1.el7.noarch 2. Which hypervisor did you use? Libvirt + KVM 2. Which storage type did you use? Ceph 3. Which networking type did you use? nova-network Logs & Configs ============== nova.conf: [DEFAULT] auth_strategy=keystone my_ip=23.10.10.4 use_neutron=true scheduler_default_filters=AllHostsFilter enabled_apis=osapi_compute,metadata firewall_driver=nova.virt.firewall.NoopFirewallDriver debug=true verbose=true rpc_backend=rabbit [api_database] connection=mysql+pymysql://nova:open1sys@controller/nova_api [cinder] os_region_name=RegionOne [database] connection=mysql+pymysql://nova:open1sys@controller/nova [glance] api_servers=http://23.10.10.254:9292 [keystone_authtoken] auth_uri = http://controller:5000 auth_url = http://controller:35357 memcached_servers = 23.10.10.4:11211,23.10.10.5:11211,23.10.10.6:11211 auth_type = password project_domain_name = default user_domain_name = default project_name = service username = nova password = open1sys [neutron] url = http://controller:9696 auth_url = http://controller:35357 auth_type = password project_domain_name = default user_domain_name = default region_name = RegionOne project_name = service username = neutron password = open1sys service_metadata_proxy = True metadata_proxy_shared_secret = open1sys [oslo_concurrency] lock_path=/var/lib/nova/tmp [oslo_messaging_rabbit] rabbit_port=5672 rabbit_hosts=23.10.10.4:5672,23.10.10.5:5672,23.10.10.6:5672 rabbit_userid=openstack rabbit_password=open1sys [vnc] vncserver_listen=$my_ip vncserver_proxyclient_address=$my_ip nova-api.log: 2017-03-24 03:14:54.618 3460 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x57a3c10>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636 2017-03-24 03:14:54.619 3460 INFO nova.osapi_compute.wsgi.server [-] 23.10.10.1 "OPTIONS / HTTP/1.0" status: 200 len: 497 time: 0.0018852 2017-03-24 03:14:56.620 3452 DEBUG nova.osapi_compute.wsgi.server [req-c89693a9-2ee6-4e90-9341-a41b19ebbee6 - - - - -] (3452) accepted ('23.10.10.1', 42466) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:867 2017-03-24 03:14:56.623 3452 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x57a3c10>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636 2017-03-24 03:14:56.624 3452 INFO nova.osapi_compute.wsgi.server [-] 23.10.10.1 "OPTIONS / HTTP/1.0" status: 200 len: 497 time: 0.0018470 2017-03-24 03:14:56.776 3452 DEBUG nova.osapi_compute.wsgi.server [req-c89693a9-2ee6-4e90-9341-a41b19ebbee6 - - - - -] (3452) accepted ('23.10.10.1', 42472) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:867 2017-03-24 03:14:57.558 3452 INFO nova.osapi_compute.wsgi.server [req-2f8b6a98-a927-4073-8322-c73c282078f2 - - - - -] 23.10.10.1 "GET /v2.1/b4c6cddfe2f74046a33784d01ded0942 HTTP/1.1" status: 404 len: 331 time: 0.7801268 2017-03-24 03:14:57.572 3452 DEBUG nova.api.openstack.wsgi [req-8d59ffbe-f6b2-4ed3-aa08-cd9ac9f0e401 - - - - -] Calling method '<bound method VersionsController.show of <nova.api.openstack.compute.versionsV21.VersionsController object at 0x7a978d0>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636 2017-03-24 03:14:57.573 3452 INFO nova.osapi_compute.wsgi.server [req-8d59ffbe-f6b2-4ed3-aa08-cd9ac9f0e401 - - - - -] 23.10.10.1 "GET /v2.1/ HTTP/1.1" status: 200 len: 717 time: 0.0126722 2017-03-24 03:14:57.950 3452 DEBUG nova.api.openstack.wsgi [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] Calling method '<bound method ServersController.detail of <nova.api.openstack.compute.servers.ServersController object at 0x794c3d0>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636 2017-03-24 03:14:57.956 3452 DEBUG nova.compute.api [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] Searching by: {'deleted': False, 'project_id': u'b4c6cddfe2f74046a33784d01ded0942'} get_all /usr/lib/python2.7/site-packages/nova/compute/api.py:2245 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] Unexpected exception in API method 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 207, in detail 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions servers = self._get_servers(req, is_detail=True) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 347, in _get_servers 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions sort_keys=sort_keys, sort_dirs=sort_dirs) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2313, in get_all 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions sort_dirs=sort_dirs) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/build_request.py", line 324, in get_by_filters 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions build_requests = cls.get_all(context) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/build_request.py", line 239, in get_all 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions db_build_reqs = cls._get_all_from_db(context) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 894, in wrapper 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return fn(*args, **kwargs) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/build_request.py", line 234, in _get_all_from_db 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions db_reqs = query.all() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2588, in all 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return list(self) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2736, in __iter__ 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return self._execute_and_instances(context) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions close_with_result=True) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions **kw) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 905, in connection 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions execution_options=execution_options) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 910, in _connection_for_bind 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions engine, execution_options) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions conn = bind.contextual_connect() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2041, in contextual_connect 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions **kwargs) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 92, in __init__ 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions self.dispatch.engine_connect(self, self.__branch) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__ 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions fn(*args, **kw) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 80, in _connect_ping_listener 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions connection.scalar(select([1])) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 844, in scalar 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return self.execute(object, *multiparams, **params).scalar() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return meth(self, multiparams, params) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return connection._execute_clauseelement(self, multiparams, params) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions compiled_sql, distilled_params 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1078, in _execute_context 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions None, None) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in _handle_dbapi_exception 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions self._autorollback() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 791, in _autorollback 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions self._root._rollback_impl() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 662, in _rollback_impl 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions self.dispatch.rollback(self) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__ 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions fn(*args, **kw) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/exc_filters.py", line 496, in pop_exc_tx 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions conn.info.pop(ROLLBACK_CAUSE_KEY, None) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 425, in info 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return self.connection.info 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 318, in connection 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions self._handle_dbapi_exception(e, None, None, None, None) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1266, in _handle_dbapi_exception 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions exc_info 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions reraise(type(exception), exception, tb=exc_tb) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 316, in connection 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return self._revalidate_connection() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 393, in _revalidate_connection 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions self.__connection = self.engine.raw_connection(_connection=self) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2104, in raw_connection 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions self.pool.unique_connection, _connection) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2080, in _wrap_pool_connect 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions util.reraise(*sys.exc_info()) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return fn() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 318, in unique_connection 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return _ConnectionFairy._checkout(self) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions fairy = _ConnectionRecord.checkout(pool) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 485, in checkout 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions rec.checkin() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__ 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions compat.reraise(exc_type, exc_value, exc_tb) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 482, in checkout 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions dbapi_connection = rec.get_connection() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 563, in get_connection 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions self.connection = self.__connect() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 607, in __connect 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions connection = self.__pool._invoke_creator(self) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return dialect.connect(*cargs, **cparams) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return self.dbapi.connect(*cargs, **cparams) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return Connection(*args, **kwargs) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 694, in __init__ 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions self.connect() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 916, in connect 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions self._request_authentication() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1124, in _request_authentication 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions auth_packet = self._read_packet() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions packet.check_error() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions err.raise_mysql_exception(self._data) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions raise errorclass(errno, errval) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections') 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions 2017-03-24 03:14:57.972 3452 INFO nova.api.openstack.wsgi [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.OperationalError'> 2017-03-24 03:14:57.973 3452 DEBUG nova.api.openstack.wsgi [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.OperationalError'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1044 2017-03-24 03:14:57.975 3452 INFO nova.osapi_compute.wsgi.server [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] 23.10.10.1 "GET /v2.1/b4c6cddfe2f74046a33784d01ded0942/servers/detail HTTP/1.1" status: 500 len: 574 time: 0.0400600 2017-03-24 03:14:58.626 3452 DEBUG nova.osapi_compute.wsgi.server [req-c89693a9-2ee6-4e90-9341-a41b19ebbee6 - - - - -] (3452) accepted ('23.10.10.1', 42502) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:867 2017-03-24 03:14:58.629 3452 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x57a3c10>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636 2017-03-24 03:14:58.631 3452 INFO nova.osapi_compute.wsgi.server [-] 23.10.10.1 "OPTIONS / HTTP/1.0" status: 200 len: 497 time: 0.0024850 2017-03-24 03:15:00.632 3452 DEBUG nova.osapi_compute.wsgi.server [req-c89693a9-2ee6-4e90-9341-a41b19ebbee6 - - - - -] (3452) accepted ('23.10.10.1', 42530) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:867 2017-03-24 03:15:00.634 3452 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x57a3c10>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636 ** Affects: nova Importance: Undecided Status: New ** Attachment added: "sosreport-ARCHS22K8144.tuc.stglabs.ibm.com-20170324032503.tar.xz" https://bugs.launchpad.net/bugs/1675720/+attachment/4844178/+files/sosreport-ARCHS22K8144.tuc.stglabs.ibm.com-20170324032503.tar.xz -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1675720 Title: NOVA report "Unexpected API Error" on Newton while run "Nova list" Status in OpenStack Compute (nova): New Bug description: Description =========== While run "nova list" to get instance list, NOVA report "[root@ARCHS22K8144 httpd]# nova list ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-e08c4a96-d986-4250-8186-7d5974b0a1b5)" randomly. Steps to reproduce ================== for i in `seq 0 100`; do echo ===========================$i; nova list; sleep 3; done Expected result =============== it should list all instances successfully. Actual result ============= most time it can list instances, but some time it report "Unexpected API Error". Environment =========== 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ # rpm -qa | grep nova openstack-nova-common-14.0.3-1.el7.noarch openstack-nova-console-14.0.3-1.el7.noarch python2-novaclient-6.0.0-1.el7.noarch openstack-nova-novncproxy-14.0.3-1.el7.noarch python-nova-14.0.3-1.el7.noarch openstack-nova-api-14.0.3-1.el7.noarch openstack-nova-conductor-14.0.3-1.el7.noarch openstack-nova-scheduler-14.0.3-1.el7.noarch 2. Which hypervisor did you use? Libvirt + KVM 2. Which storage type did you use? Ceph 3. Which networking type did you use? nova-network Logs & Configs ============== nova.conf: [DEFAULT] auth_strategy=keystone my_ip=23.10.10.4 use_neutron=true scheduler_default_filters=AllHostsFilter enabled_apis=osapi_compute,metadata firewall_driver=nova.virt.firewall.NoopFirewallDriver debug=true verbose=true rpc_backend=rabbit [api_database] connection=mysql+pymysql://nova:open1sys@controller/nova_api [cinder] os_region_name=RegionOne [database] connection=mysql+pymysql://nova:open1sys@controller/nova [glance] api_servers=http://23.10.10.254:9292 [keystone_authtoken] auth_uri = http://controller:5000 auth_url = http://controller:35357 memcached_servers = 23.10.10.4:11211,23.10.10.5:11211,23.10.10.6:11211 auth_type = password project_domain_name = default user_domain_name = default project_name = service username = nova password = open1sys [neutron] url = http://controller:9696 auth_url = http://controller:35357 auth_type = password project_domain_name = default user_domain_name = default region_name = RegionOne project_name = service username = neutron password = open1sys service_metadata_proxy = True metadata_proxy_shared_secret = open1sys [oslo_concurrency] lock_path=/var/lib/nova/tmp [oslo_messaging_rabbit] rabbit_port=5672 rabbit_hosts=23.10.10.4:5672,23.10.10.5:5672,23.10.10.6:5672 rabbit_userid=openstack rabbit_password=open1sys [vnc] vncserver_listen=$my_ip vncserver_proxyclient_address=$my_ip nova-api.log: 2017-03-24 03:14:54.618 3460 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x57a3c10>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636 2017-03-24 03:14:54.619 3460 INFO nova.osapi_compute.wsgi.server [-] 23.10.10.1 "OPTIONS / HTTP/1.0" status: 200 len: 497 time: 0.0018852 2017-03-24 03:14:56.620 3452 DEBUG nova.osapi_compute.wsgi.server [req-c89693a9-2ee6-4e90-9341-a41b19ebbee6 - - - - -] (3452) accepted ('23.10.10.1', 42466) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:867 2017-03-24 03:14:56.623 3452 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x57a3c10>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636 2017-03-24 03:14:56.624 3452 INFO nova.osapi_compute.wsgi.server [-] 23.10.10.1 "OPTIONS / HTTP/1.0" status: 200 len: 497 time: 0.0018470 2017-03-24 03:14:56.776 3452 DEBUG nova.osapi_compute.wsgi.server [req-c89693a9-2ee6-4e90-9341-a41b19ebbee6 - - - - -] (3452) accepted ('23.10.10.1', 42472) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:867 2017-03-24 03:14:57.558 3452 INFO nova.osapi_compute.wsgi.server [req-2f8b6a98-a927-4073-8322-c73c282078f2 - - - - -] 23.10.10.1 "GET /v2.1/b4c6cddfe2f74046a33784d01ded0942 HTTP/1.1" status: 404 len: 331 time: 0.7801268 2017-03-24 03:14:57.572 3452 DEBUG nova.api.openstack.wsgi [req-8d59ffbe-f6b2-4ed3-aa08-cd9ac9f0e401 - - - - -] Calling method '<bound method VersionsController.show of <nova.api.openstack.compute.versionsV21.VersionsController object at 0x7a978d0>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636 2017-03-24 03:14:57.573 3452 INFO nova.osapi_compute.wsgi.server [req-8d59ffbe-f6b2-4ed3-aa08-cd9ac9f0e401 - - - - -] 23.10.10.1 "GET /v2.1/ HTTP/1.1" status: 200 len: 717 time: 0.0126722 2017-03-24 03:14:57.950 3452 DEBUG nova.api.openstack.wsgi [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] Calling method '<bound method ServersController.detail of <nova.api.openstack.compute.servers.ServersController object at 0x794c3d0>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636 2017-03-24 03:14:57.956 3452 DEBUG nova.compute.api [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] Searching by: {'deleted': False, 'project_id': u'b4c6cddfe2f74046a33784d01ded0942'} get_all /usr/lib/python2.7/site-packages/nova/compute/api.py:2245 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] Unexpected exception in API method 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 207, in detail 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions servers = self._get_servers(req, is_detail=True) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 347, in _get_servers 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions sort_keys=sort_keys, sort_dirs=sort_dirs) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2313, in get_all 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions sort_dirs=sort_dirs) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/build_request.py", line 324, in get_by_filters 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions build_requests = cls.get_all(context) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/build_request.py", line 239, in get_all 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions db_build_reqs = cls._get_all_from_db(context) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 894, in wrapper 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return fn(*args, **kwargs) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/build_request.py", line 234, in _get_all_from_db 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions db_reqs = query.all() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2588, in all 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return list(self) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2736, in __iter__ 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return self._execute_and_instances(context) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions close_with_result=True) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions **kw) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 905, in connection 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions execution_options=execution_options) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 910, in _connection_for_bind 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions engine, execution_options) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions conn = bind.contextual_connect() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2041, in contextual_connect 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions **kwargs) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 92, in __init__ 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions self.dispatch.engine_connect(self, self.__branch) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__ 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions fn(*args, **kw) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 80, in _connect_ping_listener 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions connection.scalar(select([1])) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 844, in scalar 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return self.execute(object, *multiparams, **params).scalar() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return meth(self, multiparams, params) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return connection._execute_clauseelement(self, multiparams, params) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions compiled_sql, distilled_params 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1078, in _execute_context 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions None, None) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in _handle_dbapi_exception 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions self._autorollback() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 791, in _autorollback 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions self._root._rollback_impl() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 662, in _rollback_impl 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions self.dispatch.rollback(self) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__ 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions fn(*args, **kw) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/exc_filters.py", line 496, in pop_exc_tx 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions conn.info.pop(ROLLBACK_CAUSE_KEY, None) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 425, in info 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return self.connection.info 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 318, in connection 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions self._handle_dbapi_exception(e, None, None, None, None) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1266, in _handle_dbapi_exception 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions exc_info 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions reraise(type(exception), exception, tb=exc_tb) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 316, in connection 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return self._revalidate_connection() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 393, in _revalidate_connection 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions self.__connection = self.engine.raw_connection(_connection=self) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2104, in raw_connection 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions self.pool.unique_connection, _connection) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2080, in _wrap_pool_connect 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions util.reraise(*sys.exc_info()) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return fn() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 318, in unique_connection 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return _ConnectionFairy._checkout(self) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions fairy = _ConnectionRecord.checkout(pool) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 485, in checkout 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions rec.checkin() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__ 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions compat.reraise(exc_type, exc_value, exc_tb) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 482, in checkout 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions dbapi_connection = rec.get_connection() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 563, in get_connection 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions self.connection = self.__connect() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 607, in __connect 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions connection = self.__pool._invoke_creator(self) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return dialect.connect(*cargs, **cparams) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return self.dbapi.connect(*cargs, **cparams) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions return Connection(*args, **kwargs) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 694, in __init__ 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions self.connect() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 916, in connect 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions self._request_authentication() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1124, in _request_authentication 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions auth_packet = self._read_packet() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions packet.check_error() 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions err.raise_mysql_exception(self._data) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions raise errorclass(errno, errval) 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections') 2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions 2017-03-24 03:14:57.972 3452 INFO nova.api.openstack.wsgi [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.OperationalError'> 2017-03-24 03:14:57.973 3452 DEBUG nova.api.openstack.wsgi [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.OperationalError'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1044 2017-03-24 03:14:57.975 3452 INFO nova.osapi_compute.wsgi.server [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] 23.10.10.1 "GET /v2.1/b4c6cddfe2f74046a33784d01ded0942/servers/detail HTTP/1.1" status: 500 len: 574 time: 0.0400600 2017-03-24 03:14:58.626 3452 DEBUG nova.osapi_compute.wsgi.server [req-c89693a9-2ee6-4e90-9341-a41b19ebbee6 - - - - -] (3452) accepted ('23.10.10.1', 42502) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:867 2017-03-24 03:14:58.629 3452 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x57a3c10>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636 2017-03-24 03:14:58.631 3452 INFO nova.osapi_compute.wsgi.server [-] 23.10.10.1 "OPTIONS / HTTP/1.0" status: 200 len: 497 time: 0.0024850 2017-03-24 03:15:00.632 3452 DEBUG nova.osapi_compute.wsgi.server [req-c89693a9-2ee6-4e90-9341-a41b19ebbee6 - - - - -] (3452) accepted ('23.10.10.1', 42530) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:867 2017-03-24 03:15:00.634 3452 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x57a3c10>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1675720/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
