-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 09/07/14 15:00, Roman Podoliaka wrote: > Hi Ihar, > > AFAIU, the switch is a matter of pip install + specifying the > correct db URI in the config files. I'm not sure why you are filing > a spec in Neutron project. IMHO, this has nothing to do with > projects, but rather a purely deployment question. E.g. don't we > have PostgreSQL+psycopg2 or MySQL+pymysql deployments of OpenStack > right now?
The issue was raised in Neutron because it suffers a lot from those deadlocks, and because initially I saw the switch as local to this specific project, that would lead other projects by example and not as an enforced rule. I would be glad to put the spec in some other, better place. Do you know any? I don't know whether we have other MySQL deployments not using MySQLdb; if so, they are not configured as per official documentation. See: - - http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-database-controller.html - - http://docs.openstack.org/icehouse/install-guide/install/yum/content/basics-database-node.html - - http://docs.openstack.org/icehouse/install-guide/install/yum/content/neutron-ml2-controller-node.html If we want people to use a new module, we need to update the docs not to refer to MySQLdb. Or at least inform them that deadlocks may occur when using the library [and I think this is enough to just remove any references to the library from the documentation.] > > I think what you really want is to change the defaults we test in > the gate, which is a different problem. It's lots of different things to do: - - some work to be tracked in oslo.db (and old db code from incubator?); - - update neutron code if needed (though preliminary testing didn't reveal any specific changes for this specific project though, while others may need trivial updates); - - switch defaults in devstack; - - update documentation to refer to the new library. > > Thanks, Roman > > On Wed, Jul 9, 2014 at 2:17 PM, Ihar Hrachyshka > <ihrac...@redhat.com> wrote: Hi all, > > Multiple projects are suffering from db lock timeouts due to > deadlocks deep in mysqldb library that we use to interact with > mysql servers. In essence, the problem is due to missing eventlet > support in mysqldb module, meaning when a db lock is encountered, > the library does not yield to the next green thread, allowing other > threads to eventually unlock the grabbed lock, and instead it just > blocks the main thread, that eventually raises timeout exception > (OperationalError). > > The failed operation is not retried, leaving failing request not > served. In Nova, there is a special retry mechanism for deadlocks, > though I think it's more a hack than a proper fix. > > Neutron is one of the projects that suffer from those timeout > errors a lot. Partly it's due to lack of discipline in how we do > nested calls in l3_db and ml2_plugin code, but that's not something > to change in foreseeable future, so we need to find another > solution that is applicable for Juno. Ideally, the solution should > be applicable for Icehouse too to allow distributors to resolve > existing deadlocks without waiting for Juno. > > We've had several discussions and attempts to introduce a solution > to the problem. Thanks to oslo.db guys, we now have more or less > clear view on the cause of the failures and how to easily fix them. > The solution is to switch mysqldb to something eventlet aware. The > best candidate is probably MySQL Connector module that is an > official MySQL client for Python and that shows some (preliminary) > good results in terms of performance. > > I've posted a Neutron spec for the switch to the new client in Juno > at [1]. Ideally, switch is just a matter of several fixes to > oslo.db that would enable full support for the new driver already > supported by SQLAlchemy, plus 'connection' string modified in > service configuration files, plus documentation updates to refer to > the new official way to configure services for MySQL. The database > code won't, ideally, require any major changes, though some > adaptation for the new client library may be needed. That said, > Neutron does not seem to require any changes, though it was > revealed that there are some alembic migration rules in Keystone or > Glance that need (trivial) modifications. > > You can see how trivial the switch can be achieved for a service > based on example for Neutron [2]. > > While this is a Neutron specific proposal, there is an obvious wish > to switch to the new library globally throughout all the projects, > to reduce devops burden, among other things. My vision is that, > ideally, we switch all projects to the new library in Juno, though > we still may leave several projects for K in case any issues arise, > similar to the way projects switched to oslo.messaging during two > cycles instead of one. Though looking at how easy Neutron can be > switched to the new library, I wouldn't expect any issues that > would postpone the switch till K. > > It was mentioned in comments to the spec proposal that there were > some discussions at the latest summit around possible switch in > context of Nova that revealed some concerns, though they do not > seem to be documented anywhere. So if you know anything about it, > please comment. > > So, we'd like to hear from other projects what's your take on that > move, whether you see any issues or have concerns about it. > > Thanks for your comments, /Ihar > > [1]: https://review.openstack.org/#/c/104905/ [2]: > https://review.openstack.org/#/c/105209/ >> >> _______________________________________________ OpenStack-dev >> mailing list OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > _______________________________________________ OpenStack-dev > mailing list OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBCgAGBQJTvUEAAAoJEC5aWaUY1u57cHUIAOq+ij/rNDJv1PWmYTlaU8EB PJNdwEos9Kl2Zj37VYfJllg3oSUG8ueLXBuBXJldjgWTZZS0psNz7tmcUjTox0f/ wvpQfiFpJAXMSqRg9AYqg9dYPRAoM6PN6sSgFuNSLi1mKmRabyLafG2G5pXxWd5H C+Ku+DNpSi6qjqFnaVr7BC9Ya8wS0uhi9LGKzee/m+HjKpmJoEPO6dC1+LoDvXY6 5XFTKXk36e5WMvka+j6ZP7eS4BCfml3Gu6NjdoskLd1bEqAcIeABBO+CSRYwF1/G m2vpXX4yUG2GF5QsA4cAwvXYNo91UYtpAMhiMVzPFGQyV8rCeRYuirdd9d5Fu2E= =YvJj -----END PGP SIGNATURE----- _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
