-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 09/07/14 15:40, Sean Dague wrote: > On 07/09/2014 09:00 AM, Roman Podoliaka wrote: >> Hi Ihar, >> >> AFAIU, the switch is a matter of pip install + specifying the >> correct db URI in the config files. I'm not sure why you are >> filing a spec in Neutron project. IMHO, this has nothing to do >> with projects, but rather a purely deployment question. E.g. >> don't we have PostgreSQL+psycopg2 or MySQL+pymysql deployments of >> OpenStack right now? >> >> I think what you really want is to change the defaults we test in >> the gate, which is a different problem. > > Because this is really a *new* driver. As you can see by the > attempted run, it doesn't work with alembic given the definitions > that neutron has. So it's not like this is currently compatible > with OpenStack code.
Well, to fix that, you just need to specify raise_on_warnings=False for connection (it's default for mysqldb but not mysql-connector). I've done it in devstack patch for now, but probably it belongs to oslo.db. > >> >> Thanks, Roman >> >> On Wed, Jul 9, 2014 at 2:17 PM, Ihar Hrachyshka >> <ihrac...@redhat.com> wrote: Hi all, >> >> Multiple projects are suffering from db lock timeouts due to >> deadlocks deep in mysqldb library that we use to interact with >> mysql servers. In essence, the problem is due to missing eventlet >> support in mysqldb module, meaning when a db lock is encountered, >> the library does not yield to the next green thread, allowing >> other threads to eventually unlock the grabbed lock, and instead >> it just blocks the main thread, that eventually raises timeout >> exception (OperationalError). >> >> The failed operation is not retried, leaving failing request not >> served. In Nova, there is a special retry mechanism for >> deadlocks, though I think it's more a hack than a proper fix. >> >> Neutron is one of the projects that suffer from those timeout >> errors a lot. Partly it's due to lack of discipline in how we do >> nested calls in l3_db and ml2_plugin code, but that's not >> something to change in foreseeable future, so we need to find >> another solution that is applicable for Juno. Ideally, the >> solution should be applicable for Icehouse too to allow >> distributors to resolve existing deadlocks without waiting for >> Juno. >> >> We've had several discussions and attempts to introduce a >> solution to the problem. Thanks to oslo.db guys, we now have more >> or less clear view on the cause of the failures and how to easily >> fix them. The solution is to switch mysqldb to something eventlet >> aware. The best candidate is probably MySQL Connector module that >> is an official MySQL client for Python and that shows some >> (preliminary) good results in terms of performance. >> >> I've posted a Neutron spec for the switch to the new client in >> Juno at [1]. Ideally, switch is just a matter of several fixes to >> oslo.db that would enable full support for the new driver already >> supported by SQLAlchemy, plus 'connection' string modified in >> service configuration files, plus documentation updates to refer >> to the new official way to configure services for MySQL. The >> database code won't, ideally, require any major changes, though >> some adaptation for the new client library may be needed. That >> said, Neutron does not seem to require any changes, though it was >> revealed that there are some alembic migration rules in Keystone >> or Glance that need (trivial) modifications. >> >> You can see how trivial the switch can be achieved for a service >> based on example for Neutron [2]. >> >> While this is a Neutron specific proposal, there is an obvious >> wish to switch to the new library globally throughout all the >> projects, to reduce devops burden, among other things. My vision >> is that, ideally, we switch all projects to the new library in >> Juno, though we still may leave several projects for K in case >> any issues arise, similar to the way projects switched to >> oslo.messaging during two cycles instead of one. Though looking >> at how easy Neutron can be switched to the new library, I >> wouldn't expect any issues that would postpone the switch till >> K. >> >> It was mentioned in comments to the spec proposal that there were >> some discussions at the latest summit around possible switch in >> context of Nova that revealed some concerns, though they do not >> seem to be documented anywhere. So if you know anything about it, >> please comment. >> >> So, we'd like to hear from other projects what's your take on >> that move, whether you see any issues or have concerns about it. >> >> Thanks for your comments, /Ihar >> >> [1]: https://review.openstack.org/#/c/104905/ [2]: >> https://review.openstack.org/#/c/105209/ >>> >>> _______________________________________________ OpenStack-dev >>> mailing list OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >>> _______________________________________________ >> OpenStack-dev mailing list OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> > > > _______________________________________________ OpenStack-dev > mailing list OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBCgAGBQJTvVUaAAoJEC5aWaUY1u57likH/33btpNhAm8OjkQUj0ilBHFb HKxJDBVoWa+ioXrgYMcrwCZp8PAsE7c6jYRZW1Aaa4Ge7KeTXoAhYojF4T8p1HhQ Q3nAbxSdG1Hlq6m+6NzIGoAPxsSCgoo4hNWtyGfoBLlbpufr8QZMvXhKLObUvrZF wto3oAGXAnYu6XKjP4Y7zFMPz+t4HwIrOYNlRBlkmH+/7dzhtnVbDiJ6AsQiJ5lF +6yC5iaQEFl13TJ+5dQgTIqP5jwk9uFr/nhakj+4hbZWpAjXf0fqY+jNaUL3dRDh hFuy4f0sRlCgSiq7x6Rt9NO5UYnwSd7yHLjZBDyzCdOQrdweDSOH6XB26K3yJr0= =kSGB -----END PGP SIGNATURE----- _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
