Hi all, Not sure what issues you are talking about, but I just replaced "mysql" with "mysql+mysqlconnector" in my db connection string in neutron.conf and "neutron-db-manage upgrade head" worked like a charm for an empty schema.
Ihar, could please elaborate on what changes to oslo.db are needed? (as an oslo.db developer I'm very interested in this part :) ) Thanks, Roman On Wed, Jul 9, 2014 at 5:43 PM, Ihar Hrachyshka <ihrac...@redhat.com> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > On 09/07/14 15:40, Sean Dague wrote: >> On 07/09/2014 09:00 AM, Roman Podoliaka wrote: >>> Hi Ihar, >>> >>> AFAIU, the switch is a matter of pip install + specifying the >>> correct db URI in the config files. I'm not sure why you are >>> filing a spec in Neutron project. IMHO, this has nothing to do >>> with projects, but rather a purely deployment question. E.g. >>> don't we have PostgreSQL+psycopg2 or MySQL+pymysql deployments of >>> OpenStack right now? >>> >>> I think what you really want is to change the defaults we test in >>> the gate, which is a different problem. >> >> Because this is really a *new* driver. As you can see by the >> attempted run, it doesn't work with alembic given the definitions >> that neutron has. So it's not like this is currently compatible >> with OpenStack code. > > Well, to fix that, you just need to specify raise_on_warnings=False > for connection (it's default for mysqldb but not mysql-connector). > I've done it in devstack patch for now, but probably it belongs to > oslo.db. > >> >>> >>> Thanks, Roman >>> >>> On Wed, Jul 9, 2014 at 2:17 PM, Ihar Hrachyshka >>> <ihrac...@redhat.com> wrote: Hi all, >>> >>> Multiple projects are suffering from db lock timeouts due to >>> deadlocks deep in mysqldb library that we use to interact with >>> mysql servers. In essence, the problem is due to missing eventlet >>> support in mysqldb module, meaning when a db lock is encountered, >>> the library does not yield to the next green thread, allowing >>> other threads to eventually unlock the grabbed lock, and instead >>> it just blocks the main thread, that eventually raises timeout >>> exception (OperationalError). >>> >>> The failed operation is not retried, leaving failing request not >>> served. In Nova, there is a special retry mechanism for >>> deadlocks, though I think it's more a hack than a proper fix. >>> >>> Neutron is one of the projects that suffer from those timeout >>> errors a lot. Partly it's due to lack of discipline in how we do >>> nested calls in l3_db and ml2_plugin code, but that's not >>> something to change in foreseeable future, so we need to find >>> another solution that is applicable for Juno. Ideally, the >>> solution should be applicable for Icehouse too to allow >>> distributors to resolve existing deadlocks without waiting for >>> Juno. >>> >>> We've had several discussions and attempts to introduce a >>> solution to the problem. Thanks to oslo.db guys, we now have more >>> or less clear view on the cause of the failures and how to easily >>> fix them. The solution is to switch mysqldb to something eventlet >>> aware. The best candidate is probably MySQL Connector module that >>> is an official MySQL client for Python and that shows some >>> (preliminary) good results in terms of performance. >>> >>> I've posted a Neutron spec for the switch to the new client in >>> Juno at [1]. Ideally, switch is just a matter of several fixes to >>> oslo.db that would enable full support for the new driver already >>> supported by SQLAlchemy, plus 'connection' string modified in >>> service configuration files, plus documentation updates to refer >>> to the new official way to configure services for MySQL. The >>> database code won't, ideally, require any major changes, though >>> some adaptation for the new client library may be needed. That >>> said, Neutron does not seem to require any changes, though it was >>> revealed that there are some alembic migration rules in Keystone >>> or Glance that need (trivial) modifications. >>> >>> You can see how trivial the switch can be achieved for a service >>> based on example for Neutron [2]. >>> >>> While this is a Neutron specific proposal, there is an obvious >>> wish to switch to the new library globally throughout all the >>> projects, to reduce devops burden, among other things. My vision >>> is that, ideally, we switch all projects to the new library in >>> Juno, though we still may leave several projects for K in case >>> any issues arise, similar to the way projects switched to >>> oslo.messaging during two cycles instead of one. Though looking >>> at how easy Neutron can be switched to the new library, I >>> wouldn't expect any issues that would postpone the switch till >>> K. >>> >>> It was mentioned in comments to the spec proposal that there were >>> some discussions at the latest summit around possible switch in >>> context of Nova that revealed some concerns, though they do not >>> seem to be documented anywhere. So if you know anything about it, >>> please comment. >>> >>> So, we'd like to hear from other projects what's your take on >>> that move, whether you see any issues or have concerns about it. >>> >>> Thanks for your comments, /Ihar >>> >>> [1]: https://review.openstack.org/#/c/104905/ [2]: >>> https://review.openstack.org/#/c/105209/ >>>> >>>> _______________________________________________ OpenStack-dev >>>> mailing list OpenStack-dev@lists.openstack.org >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>>> > _______________________________________________ >>> OpenStack-dev mailing list OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >>> >> >> >> _______________________________________________ OpenStack-dev >> mailing list OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > -----BEGIN PGP SIGNATURE----- > Version: GnuPG/MacGPG2 v2.0.22 (Darwin) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQEcBAEBCgAGBQJTvVUaAAoJEC5aWaUY1u57likH/33btpNhAm8OjkQUj0ilBHFb > HKxJDBVoWa+ioXrgYMcrwCZp8PAsE7c6jYRZW1Aaa4Ge7KeTXoAhYojF4T8p1HhQ > Q3nAbxSdG1Hlq6m+6NzIGoAPxsSCgoo4hNWtyGfoBLlbpufr8QZMvXhKLObUvrZF > wto3oAGXAnYu6XKjP4Y7zFMPz+t4HwIrOYNlRBlkmH+/7dzhtnVbDiJ6AsQiJ5lF > +6yC5iaQEFl13TJ+5dQgTIqP5jwk9uFr/nhakj+4hbZWpAjXf0fqY+jNaUL3dRDh > hFuy4f0sRlCgSiq7x6Rt9NO5UYnwSd7yHLjZBDyzCdOQrdweDSOH6XB26K3yJr0= > =kSGB > -----END PGP SIGNATURE----- > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
