Excerpts from jeblair's message of 2014-03-26 13:30:28 -0700: > Hi, > > I recently started looking into running the unit tests with MySQL. I > pushed the start of a change here to do that: > > https://review.openstack.org/#/c/82872/ > > In particular, that change implements something we've wanted to do for a > long time elsewhere in OpenStack -- create a dedicated MySQL database > for each test so that tests can safely run in parallel. However, that > has uncovered the following issues: > > * The tests do not use alembic, except for the migration test. This > could lead to variations in what the unit tests are testing against > as compared to production. > > * The model does not specify an engine for MySQL (this is a difference > between the model and the alembic migrations). So when the unit tests > are run on MySQL, they may end up on MyISAM (they do in the gate). >
Why MyISAM? The default has been InnoDB since MySQL 5.5. > * The two errors that show up in the test results for my patch relate > to how timestamps are stored differently between MySQL and SQLite. > Isn't this a bug in sqlalchemy? I'd expect any data type that is a common name in the two databases (and thus the same in the model definition) to produce the same data in the python representation. > * The errors that you can not see in the test report from Jenkins but > would if you ran in an environment with InnoDB as the default engine > for MySQL are several instances of foreign key constraint errors. > Side note: IMHO: Just drop the FK constraints if you want the app to be scalable. FK's just add extra reads for every write, when you could be doing that verification / orphan clean up offline later on. > I think it is desirable to run the unit tests with MySQL largely because > I think we should be testing it in the way we intend to use it in > production. Note especially the last point above -- our unit tests have > significant problems with MySQL. That suggests one of two things to me: > either the tests aren't testing real behavior, or our system is actually > broken in production. Neither one seems like a good situation. > Wouldn't integration tests be the place to find production vs. code path issues? Tying unit tests to the creation of mysql databases introduces a really big piece of machinery into the test fixture setup, which you want to be fast and lean so developers can iterate on it while working. > My inclination would be to use MySQL for unit tests, use alembic to > create the schema, and fix the errors currently uncovered by this. > > It's not going to be a trivial change, but I think it's worth doing. > Before embarking on it, I'd like to see if we can get consensus. We can > talk about it here or possibly at the meeting tomorrow. > > Ruslan has commented in that review with the following: > > > But, I think that we should consider work done in major OpenStack > > projects. They don't use real MySQL to run unit-tests. Here is what > > the do (or plan to do): > > > > * Run unit-tests on SQLite. DB schema is created from SQLA metadata > > * Run DB migration tests on MySQL and Postgres > > * Run tests to compare SQLA metadata and DB migrations to make sure > > that they're in sync. Here is WIP patch > > https://review.openstack.org/#/c/ > > > > I'm not opposed to this direction, I just want to make sure we're > > aligned with the common practices in OpenStack > > (That last link doesn't seem to be correct.) > > That's a good point, but I personally find testing as close to > production as possible desirable -- MySQL and SQLite are not compatible > so we shouldn't pretend that they are. If we prove this is a viable > strategy, then I think we should recommend OpenStack projects adopt it > as well. > I just wonder if SQLAlchemy is intended to "smooth out" the differences between the two in such a way where you can at least be confident that your code is doing what you think it is. _______________________________________________ OpenStack-Infra mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
