On Thu, May 3, 2012 at 10:01 AM, Gary Kotton <gkot...@redhat.com> wrote:
> ** > On 05/03/2012 07:15 PM, Dan Wendlandt wrote: > > Hi Gary, > > Great to see all the effort on improving usability + consistency. > > On Thu, May 3, 2012 at 7:36 AM, Gary Kotton <gkot...@redhat.com> wrote: > > > > > > Hi, > > I have taken a more thorough look at the database implementations. On > the plugin side all of the plugins make use of the common database code > (../quantum/quantum/db/*). > > > Yes, or at least, any plugin that uses a database uses the common database > code. Some plugins don't have a local database at all, as they rely on an > external system (NVP is the only example in the repo, but I've heard of > other people developing plugins that don't use a local DB). > > > > > Today I tested a number of scenarios with connectivity to the server. > This works well with the linuxbridge and the openvswitch plugins. Taking > into account that we working on the scalability of the agents ( > https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms), I > am pretty convinced that a large part of the blueprint is redundant. > Nonetheless I still think we need to address > > 1. Common configuration for the database strings (similarly to those in > OpenStack) > > > Agreed on this one. > > > > > 2. If the Quantum service starts and the database is down. Quantum > service terminates. Output: > > [f17-dev@localhost ~]$ quantum list_nets default > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py", > line 237, in list_nets_v11 > > res = client.list_networks(**filters) > > File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", > line 145, in with_params > > ret = self.function(instance, *args) > > File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", > line 522, in list_networks > > return self.get(self.networks_path, params=filters) > > File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", > line 379, in get > > headers=headers, params=params) > > File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", > line 371, in retry_request > > raise exceptions.ConnectionFailed(reason=_("Maximum attempts > reached")) > > ConnectionFailed: Connection to quantum failed: Maximum attempts > reached > > > Does the service terminate? Or is it just that each request fails? The > former would definitely be a problem, but the stack trace at least suggests > that the service actually catches the error. > > The service terminates. I'll use the fixes below and get back to you if it > resolved. > Perhaps our setup is different, as I am not seeing the service (i.e., quantum-server) terminate if the database is down. Below I list networks, stop mysql, confirm that mysql is down, list networks with quantum client (get client error), use screen to confirm that service continues to run despite error, restart mysql, list networks with quantum client again demonstrating that quantum-server is again accessing the restarted database. Are you seeing something else for the same pattern of events? Dan root@devstack-multi1:~# quantum list_nets default Virtual Networks for Tenant default Network ID: f6513015-2ba5-443b-8055-a2809cd34c20 root@devstack-multi1:~# /etc/init.d/mysql stop root@devstack-multi1:~# mysql -uroot -pnova ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) root@devstack-multi1:~# su stack stack@devstack-multi1:/root$ screen -x stack [1]+ Stopped screen -x stack stack@devstack-multi1:/root$ quantum list_nets default Traceback (most recent call last): File "/opt/stack/python-quantumclient/quantum/client/cli_lib.py", line 237, in list_nets_v11 res = client.list_networks(**filters) File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line 145, in with_params ret = self.function(instance, *args) File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line 522, in list_networks return self.get(self.networks_path, params=filters) File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line 379, in get headers=headers, params=params) File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line 364, in retry_request headers=headers, params=params) File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line 305, in do_request self._handle_fault_response(status_code, data) File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line 228, in _handle_fault_response EXCEPTION_HANDLERS[self.version](status_code, des_error_body) File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line 119, in exception_handler_v11 raise exceptions.QuantumClientException(message=msg) QuantumClientException: 500-{'message': 'Traceback (most recent call last):\n File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 336, in handle_one_response\n result = self.application(self.environ, start_response)\n File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 203, in __call__\n return app(environ, start_response)\n File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__\n return resp(environ, start_response)\n File "/usr/lib/pymodules/python2.7/routes/middleware.py", line 131, in __call__\n response = self.app(environ, start_response)\n File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__\n return resp(environ, start_response)\n File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__\n return resp(environ, start_response)\n File "/usr/lib/pymodules/python2.7/routes/middleware.py", line 131, in __call__\n response = self.app(environ, start_response)\n File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__\n return resp(environ, start_response)\n File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 208, in call_func\n return self.func(req, *args, **kwargs)\n File "/opt/stack/quantum/quantum/wsgi.py", line 748, in __call__\n action_result = self.dispatch(request, action, args)\n File "/opt/stack/quantum/quantum/wsgi.py", line 780, in dispatch\n return controller_method(request=request, **action_args)\n File "/opt/stack/quantum/quantum/api/api_common.py", line 108, in the_func\n return func(*args, **kwargs)\n File "/opt/stack/quantum/quantum/api/networks.py", line 101, in index\n return self._items(request, tenant_id)\n File "/opt/stack/quantum/quantum/api/networks.py", line 84, in _items\n filter_opts=filter_opts)\n File "/opt/stack/quantum/quantum/plugins/openvswitch/ovs_quantum_plugin.py", line 122, in get_all_networks\n for x in db.network_list(tenant_id):\n File "/opt/stack/quantum/quantum/db/api.py", line 131, in network_list\n filter_by(tenant_id=tenant_id).\\\n File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2066, in all\n return list(self)\n File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2176, in __iter__\n return self._execute_and_instances(context)\n File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2189, in _execute_and_instances\n close_with_result=True)\n File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2180, in _connection_from_session\n **kw)\n File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 729, in connection\n close_with_result=close_with_result)\n File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 735, in _connection_for_bind\n return engine.contextual_connect(**kwargs)\n File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2476, in contextual_connect\n self.pool.connect(),\n File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py", line 224, in connect\n return _ConnectionFairy(self).checkout()\n File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py", line 466, in checkout\n self.connection = self._connection_record.get_connection()\n File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py", line 300, in get_connection\n self.connection = self.__connect()\n File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py", line 330, in __connect\n connection = self.__pool._creator()\n File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/engine/strategies.py", line 80, in connect\n return dialect.connect(*cargs, **cparams)\n File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/engine/default.py", line 285, in connect\n return self.dbapi.connect(*cargs, **cparams)\n File "/usr/lib/pymodules/python2.7/MySQLdb/__init__.py", line 81, in Connect\n return Connection(*args, **kwargs)\n File "/usr/lib/pymodules/python2.7/MySQLdb/connections.py", line 187, in __init__\n super(Connection, self).__init__(*args, **kwargs2)\nOperationalError: (OperationalError) (2002, "Can\'t connect to local MySQL server through socket \'/var/run/mysqld/mysqld.sock\' (2)") None None\n'} stack@devstack-multi1:/root$ screen -x stack [2]+ Stopped screen -x stack stack@devstack-multi1:/root$ sudo service mysql start mysql start/running, process 27677 stack@devstack-multi1:/root$ quantum list_nets default Virtual Networks for Tenant default Network ID: f6513015-2ba5-443b-8055-a2809cd34c20 > > One improvement we definitely need is to hide such errors from the tenant > (who should just get generic errors). We had a bug filed on that already, > and Guilherme proposed a patch for it yesterday: > https://review.openstack.org/#/c/7019/ . Perhaps that is what you're > suggesting below? > > Brilliant - this addresses part of it (saved a few -1's for me :)) > > > Thanks! > > dan > > > > > 3. Error handling - for example when using the CLI and the DB is down we > get the attached output. It would be better if the trace was looged and the > user would receive the specific error message. > > [f17-dev@localhost ~]$ quantum list_nets default > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py", > line 237, in list_nets_v11 > > res = client.list_networks(**filters) > > File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", > line 145, in with_params > > ret = self.function(instance, *args) > > File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", > line 522, in list_networks > > ... > > raise exceptions.QuantumClientException(message=msg) > > QuantumClientException: 500-{'message': 'Traceback (most recent call > last):\n File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line > 336, in handle_one_response\n result = > self.application(self.environ, start_response)\n File > "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in __call__\n > return app(environ, start_response)\n File > "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n > return resp(environ, start_response)\n File > "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in > __call__\n response = self.app(environ, start_response)\n File > "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n > return resp(environ, start_response)\n File > "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n > return resp(environ, start_response)\n File > "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in > __call__\n response = self.app(environ, start_response)\n File > "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n > return resp(environ, start_response)\n File > "/usr/lib/python2.7/site-packages/webob/dec.py", line 147, in __call__\n > resp = self.call_func(req, *args, **self.kwargs)\n File > "/usr/lib/python2.7/site-packages/webob/dec.py", line 210, in call_func\n > return self.func(req, *args, **kwargs)\n File > "/usr/lib/python2.7/site-packages/quantum/wsgi.py", line 748, in __call__\n > action_result = self.dispatch(request, action, args)\n File > "/usr/lib/python2.7/site-packages/quantum/wsgi.py", line 780, in dispatch\n > return controller_method(request=request, **action_args)\n File > "/usr/lib/python2.7/site-packages/quantum/api/api_common.py", line 108, in > the_func\n return func(*args, **kwargs)\n File > "/usr/lib/python2.7/site-packages/quantum/api/networks.py", line 101, in > index\n return self._items(request, tenant_id)\n File > "/usr/lib/python2.7/site-packages/quantum/api/networks.py", line 84, in > _items\n filter_opts=filter_opts)\n File > "/usr/lib/python2.7/site-packages/quantum/plugins/linuxbridge/LinuxBridgePlugin.py", > line 71, in get_all_networks\n networks_list = > db.network_list(tenant_id)\n File > "/usr/lib/python2.7/site-packages/quantum/db/api.py", line 131, in > network_list\n filter_by(tenant_id=tenant_id).\\\n File > "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2066, in > all\n return list(self)\n File > "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2176, in > __iter__\n return self._execute_and_instances(context)\n File > "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2189, in > _execute_and_instances\n close_with_result=True)\n File > "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2180, in > _connection_from_session\n **kw)\n File > "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 729, > in connection\n close_with_result=close_with_result)\n File > "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 735, > in _connection_for_bind\n return engine.contextual_connect(**kwargs)\n > File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line > 2476, in contextual_connect\n self.pool.connect(),\n File > "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 224, in > connect\n return _ConnectionFairy(self).checkout()\n File > "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 466, in > checkout\n self.connection = self._connection_record.get_connection()\n > File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 300, in > get_connection\n self.connection = self.__connect()\n File > "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 330, in > __connect\n connection = self.__pool._creator()\n File > "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line > 80, in connect\n return dialect.connect(*cargs, **cparams)\n File > "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line > 285, in connect\n return self.dbapi.connect(*cargs, **cparams)\n File > "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in > Connect\n return Connection(*args, **kwargs)\n File > "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187, in > __init__\n super(Connection, self).__init__(*args, > **kwargs2)\nOperationalError: (OperationalError) (2003, "Can\'t connect to > MySQL server on \'localhost.localdomain\' (111)") None None\n'} > > > > Any thought and or comments. Any objections if I update the blueprint to > deal with the the above items. > > Thanks > > Gary > > > > > > -- > > Mailing list: https://launchpad.net/~netstack > > Post to : netstack@lists.launchpad.net > > Unsubscribe : https://launchpad.net/~netstack > > More help : https://help.launchpad.net/ListHelp > > > > > -- > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Dan Wendlandt > Nicira, Inc: www.nicira.com > twitter: danwendlandt > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > > -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Dan Wendlandt Nicira, Inc: www.nicira.com twitter: danwendlandt ~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- Mailing list: https://launchpad.net/~netstack Post to : netstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~netstack More help : https://help.launchpad.net/ListHelp
