On 08/27/2013 04:32 AM, Boris Pavlovic wrote:
Jay,
I should probably share to you about our work around DB.
Migrations should be run only in production and only for production
backends (e.g. psql and mysql)
In tests we should use Schemas created by Models
(BASE.metadata.create_all())
Agree on both.
We are not able to use in this approach in moment because we don't have
any mechanism to check that MODELS and SCHEMAS are EQUAL.
And actually MODELS and SCHEMAS are DIFFERENT.
Sorry, I don't understand the connection... how does not having a
codified way of determining the difference between model and schema
(BTW, this does exist in sqlalchemy-migrate... look at the
compare_model_to_db method) not allow you to use metadata.create_all()
in tests or mean that you can't run migrations only in production?
E.g. in Celiometer we have BP that syncs models and migration
https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-db-sync-models-with-migrations
(in other projects we are doing the same)
And also we are working around (oslo) generic tests that checks that
models and migrations are equal:
https://review.openstack.org/#/c/42307/
OK, cool.
So in our roadmap (in this case is):
1) Soft switch to alembic (with code that allows to have sqla-migrate
and alembic migration in the same time)
I don't see the point in this at all... I would rather see patches that
just switch to Alembic and get rid of SQLAlchemy-migrate. Create an
initial Alembic migration that has the last state of the database schema
under SQLAlchemy-migrate... and then delete SA-Migrate.
2) Sync Models and Migrations (fix DB schemas also)
3) Add from oslo generic test that checks all this stuff
4) Use BASE.create_all() for Schema creation instead of migrations.
This is already done in some projects, IIRC... (Glance used to be this
way, at least)
But in OpenStack is not so simple to implement such huge changes, so it
take some time=)
Best regards,
Boris Pavlovic
---
Mirantis Inc.
On Tue, Aug 27, 2013 at 12:02 AM, Jay Pipes <[email protected]
<mailto:[email protected]>> wrote:
On 08/26/2013 03:40 PM, Herndon, John Luke (HPCS - Ft. Collins) wrote:
Jay -
It looks there is an error in the migration script that causes
it to abort:
AttributeError: 'ForeignKeyConstraint' object has no attribute
'drop'
My guess is the migration runs on the first test, creates event
types
table fine, but exits with the above error, so migration is not
"complete". Thus every subsequent test tries to migrate the db, and
notices that event types already exists.
I'd corrected that particular mistake and pushed an updated
migration script.
Best,
-jay
-john
On 8/26/13 1:15 PM, "Jay Pipes" <[email protected]
<mailto:[email protected]>> wrote:
I just noticed that every single test case for SQL-driver
storage is
executing every single migration upgrade before every single
test case
run:
https://github.com/openstack/__ceilometer/blob/master/__ceilometer/tests/db.py
<https://github.com/openstack/ceilometer/blob/master/ceilometer/tests/db.py>
#L46
https://github.com/openstack/__ceilometer/blob/master/__ceilometer/storage/imp
<https://github.com/openstack/ceilometer/blob/master/ceilometer/storage/imp>
l_sqlalchemy.py#L153
instead of simply creating a new database schema from the
models in the
current source code base using a call to
sqlalchemy.MetaData.create___all().
This results in re-running migrations over and over again,
instead of
having dedicated migration tests that would test each migration
individually, as is the case in projects like Glance...
Is this intentional?
Best,
-jay
On 08/26/2013 02:59 PM, Sandy Walsh wrote:
I'm getting the same problem with a different migration
(mine is
complaining that a column already exists)
http://paste.openstack.org/__show/44512/
<http://paste.openstack.org/show/44512/>
I've compared it to the other migrations and it seems fine.
-S
On 08/26/2013 02:34 PM, Jay Pipes wrote:
Hey all,
I'm trying to figure out what is going wrong with my
code for this
patch:
https://review.openstack.org/__41316
<https://review.openstack.org/41316>
I had previously added a sqlalchemy-migrate
migration script to add an
event_type table, and had that working, but then was
asked to instead
use Alembic for migrations. So, I removed the
sqlalchemy-migrate
migration file and added an Alembic migration [1].
Unfortunately, I am getting the following error when
running tests:
OperationalError: (OperationalError) table
event_type already exists
u'\nCREATE TABLE event_type (\n\tid INTEGER NOT
NULL, \n\t"desc"
VARCHAR(255), \n\tPRIMARY KEY (id), \n\tUNIQUE
("desc")\n)\n\n' ()
The migration adds the event_type table. I've seen
this error occur
before when using SQLite due to SQLite's ALTER TABLE
statement not
allowing the rename of a column. In the
sqlalchemy-migrate migration, I
had a specialized SQLite migration upgrade [2] and
downgrade [3]
script,
but I'm not sure how I am supposed to handle this in
Alembic. Could
someone help me out?
Thanks,
-jay
[1]
https://review.openstack.org/#__/c/41316/16/ceilometer/__storage/sqlalchemy/
<https://review.openstack.org/#/c/41316/16/ceilometer/storage/sqlalchemy/>
alembic/versions/49036daaaafd___add_event_types.py
[2]
https://review.openstack.org/#__/c/41316/14/ceilometer/__storage/sqlalchemy/
<https://review.openstack.org/#/c/41316/14/ceilometer/storage/sqlalchemy/>
migrate_repo/versions/013___sqlite_upgrade.sql
[3]
https://review.openstack.org/#__/c/41316/14/ceilometer/__storage/sqlalchemy/
<https://review.openstack.org/#/c/41316/14/ceilometer/storage/sqlalchemy/>
migrate_repo/versions/013___sqlite_downgrade.sql
_________________________________________________
OpenStack-dev mailing list
[email protected].__org
<mailto:[email protected]>
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
_________________________________________________
OpenStack-dev mailing list
[email protected].__org
<mailto:[email protected]>
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
_________________________________________________
OpenStack-dev mailing list
[email protected].__org
<mailto:[email protected]>
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
_________________________________________________
OpenStack-dev mailing list
[email protected].__org
<mailto:[email protected]>
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
