Hello,
Please see my answers inline and my previous email regarding this topic:
Hello,
This conversion is actually quite simple. We are currently working to
support alembic migrations in ceilometer:
https://blueprints.launchpad.net/ceilometer/+spec/convert-to-alembic
For now we agreed that conversion process will be the following:
1) Create folder for alembic migrations and configure engine correctly
2) Run alembic migrations after sqlalchemy-migrate (alembic creates a
separate stamp table by default)
3) Create new migrations in alembic
4) Sync model with migrations* -
https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-db-sync-models-with-migrations
5) Remove old migration files after the end of support period (2
releases)
* Is needed to remove the need of base migration so the clean database
could be created from models directly without the need of migrations.
This allows to simply drop old migrations without the need to compact
them to one big migration (133_folsom.py in nova for example)
Please share your thoughts about proposed process.
Regards,
Alexei Kornienko
Regards,
Alexei Kornienko
On 08/27/2013 07:30 PM, Jay Pipes wrote:
On 08/27/2013 04:32 AM, Boris Pavlovic wrote:
Jay,
I should probably share to you about our work around DB.
Migrations should be run only in production and only for production
backends (e.g. psql and mysql)
In tests we should use Schemas created by Models
(BASE.metadata.create_all())
Agree on both.
We are not able to use in this approach in moment because we don't have
any mechanism to check that MODELS and SCHEMAS are EQUAL.
And actually MODELS and SCHEMAS are DIFFERENT.
Sorry, I don't understand the connection... how does not having a
codified way of determining the difference between model and schema
(BTW, this does exist in sqlalchemy-migrate... look at the
compare_model_to_db method) not allow you to use metadata.create_all()
in tests or mean that you can't run migrations only in production?
As Boris said we'll use 2 completely different ways to create DB schema
in production and test environment. Cause of this we won't be able to
guarantee that code is correct unless we'll have a dedicated test that
will assure that we work with the same DB schema in both envs.
E.g. in Celiometer we have BP that syncs models and migration
https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-db-sync-models-with-migrations
(in other projects we are doing the same)
And also we are working around (oslo) generic tests that checks that
models and migrations are equal:
https://review.openstack.org/#/c/42307/
OK, cool.
So in our roadmap (in this case is):
1) Soft switch to alembic (with code that allows to have sqla-migrate
and alembic migration in the same time)
I don't see the point in this at all... I would rather see patches
that just switch to Alembic and get rid of SQLAlchemy-migrate. Create
an initial Alembic migration that has the last state of the database
schema under SQLAlchemy-migrate... and then delete SA-Migrate.
It's not that simple as it seems. Please take into account that we have
to keep SA-migrate + migrations during maintenance period for all
projects. Cause of this we have to go the long way and keep both engines
running before we'll be able to completely remove SA-migrate.
2) Sync Models and Migrations (fix DB schemas also)
3) Add from oslo generic test that checks all this stuff
4) Use BASE.create_all() for Schema creation instead of migrations.
This is already done in some projects, IIRC... (Glance used to be this
way, at least)
But in OpenStack is not so simple to implement such huge changes, so it
take some time=)
Best regards,
Boris Pavlovic
---
Mirantis Inc.
On Tue, Aug 27, 2013 at 12:02 AM, Jay Pipes <[email protected]
<mailto:[email protected]>> wrote:
On 08/26/2013 03:40 PM, Herndon, John Luke (HPCS - Ft. Collins)
wrote:
Jay -
It looks there is an error in the migration script that causes
it to abort:
AttributeError: 'ForeignKeyConstraint' object has no attribute
'drop'
My guess is the migration runs on the first test, creates event
types
table fine, but exits with the above error, so migration is not
"complete". Thus every subsequent test tries to migrate the
db, and
notices that event types already exists.
I'd corrected that particular mistake and pushed an updated
migration script.
Best,
-jay
-john
On 8/26/13 1:15 PM, "Jay Pipes" <[email protected]
<mailto:[email protected]>> wrote:
I just noticed that every single test case for SQL-driver
storage is
executing every single migration upgrade before every single
test case
run:
https://github.com/openstack/__ceilometer/blob/master/__ceilometer/tests/db.py
<https://github.com/openstack/ceilometer/blob/master/ceilometer/tests/db.py>
#L46
https://github.com/openstack/__ceilometer/blob/master/__ceilometer/storage/imp
<https://github.com/openstack/ceilometer/blob/master/ceilometer/storage/imp>
l_sqlalchemy.py#L153
instead of simply creating a new database schema from the
models in the
current source code base using a call to
sqlalchemy.MetaData.create___all().
This results in re-running migrations over and over again,
instead of
having dedicated migration tests that would test each
migration
individually, as is the case in projects like Glance...
Is this intentional?
Best,
-jay
On 08/26/2013 02:59 PM, Sandy Walsh wrote:
I'm getting the same problem with a different migration
(mine is
complaining that a column already exists)
http://paste.openstack.org/__show/44512/
<http://paste.openstack.org/show/44512/>
I've compared it to the other migrations and it seems
fine.
-S
On 08/26/2013 02:34 PM, Jay Pipes wrote:
Hey all,
I'm trying to figure out what is going wrong with my
code for this
patch:
https://review.openstack.org/__41316
<https://review.openstack.org/41316>
I had previously added a sqlalchemy-migrate
migration script to add an
event_type table, and had that working, but then was
asked to instead
use Alembic for migrations. So, I removed the
sqlalchemy-migrate
migration file and added an Alembic migration [1].
Unfortunately, I am getting the following error when
running tests:
OperationalError: (OperationalError) table
event_type already exists
u'\nCREATE TABLE event_type (\n\tid INTEGER NOT
NULL, \n\t"desc"
VARCHAR(255), \n\tPRIMARY KEY (id), \n\tUNIQUE
("desc")\n)\n\n' ()
The migration adds the event_type table. I've seen
this error occur
before when using SQLite due to SQLite's ALTER TABLE
statement not
allowing the rename of a column. In the
sqlalchemy-migrate migration, I
had a specialized SQLite migration upgrade [2] and
downgrade [3]
script,
but I'm not sure how I am supposed to handle this in
Alembic. Could
someone help me out?
Thanks,
-jay
[1]
https://review.openstack.org/#__/c/41316/16/ceilometer/__storage/sqlalchemy/
<https://review.openstack.org/#/c/41316/16/ceilometer/storage/sqlalchemy/>
alembic/versions/49036daaaafd___add_event_types.py
[2]
https://review.openstack.org/#__/c/41316/14/ceilometer/__storage/sqlalchemy/
<https://review.openstack.org/#/c/41316/14/ceilometer/storage/sqlalchemy/>
migrate_repo/versions/013___sqlite_upgrade.sql
[3]
https://review.openstack.org/#__/c/41316/14/ceilometer/__storage/sqlalchemy/
<https://review.openstack.org/#/c/41316/14/ceilometer/storage/sqlalchemy/>
migrate_repo/versions/013___sqlite_downgrade.sql
_________________________________________________
OpenStack-dev mailing list
[email protected].__org
<mailto:[email protected]>
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
_________________________________________________
OpenStack-dev mailing list
[email protected].__org
<mailto:[email protected]>
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
_________________________________________________
OpenStack-dev mailing list
[email protected].__org
<mailto:[email protected]>
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
_________________________________________________
OpenStack-dev mailing list
[email protected].__org
<mailto:[email protected]>
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
