On 08/25/2016 01:13 PM, Steve Martinelli wrote:
The keystone team is pursuing a trigger-based approach to support
rolling, zero-downtime upgrades. The proposed operator experience is
documented here:
http://docs.openstack.org/developer/keystone/upgrading.html
This differs from Nova and Neutron's approaches to solve for rolling
upgrades (which use oslo.versionedobjects), however Keystone is one of
the few services that doesn't need to manage communication between
multiple releases of multiple service components talking over the
message bus (which is the original use case for oslo.versionedobjects,
and for which it is aptly suited). Keystone simply scales horizontally
and every node talks directly to the database.
Hi Steve -
I'm a strong proponent of looking into the use of triggers to smooth
upgrades between database versions. Even in the case of projects
using versioned objects, it still means a SQL layer has to include
functionality for both versions of a particular schema change which
itself is awkward. I'm also still a little worried that not every case
of this can be handled by orchestration at the API level, and not as a
single SQL layer method that integrates both versions of a schema change.
Using triggers would resolve the issue of SQL-specific application code
needing to refer to two versions of a schema at once, at least for those
areas where triggers and SPs can handle it. In the "ideal", it means
all the Python code can just refer to one version of a schema, and nuts
and bolts embedded into database migrations would handle all the
movement between schema versions, including the phase between expand and
contract. Not that I think the "ideal" is ever going to be realized
100%, but maybe in some / many places, this can work.
So if Keystone wants to be involved in paving the way for working with
triggers, IMO this would benefit other projects in that they could
leverage this kind of functionality in those places where it makes sense.
The problem of "zero downtime database migrations" is an incredibly
ambitious goal and I think it would be wrong to exclude any one
particular technique in pursuing this. A real-world success story would
likely integrate many different techniques as they apply to specific
scenarios, and triggers and SPs IMO are a really major one which I
believe can be supported.
Database triggers are obviously a new challenge for developers to write,
honestly challenging to debug (being side effects), and are made even
more difficult by having to hand write triggers for MySQL, PostgreSQL,
and SQLite independently (SQLAlchemy offers no assistance in this case),
as seen in this patch:
So I would also note that we've been working on the availability of
triggers and stored functions elsewhere, a very raw patch that is to be
largely rolled into oslo.db is here:
https://review.openstack.org/#/c/314054/
This patch makes use of an Alembic pattern called "replaceable object",
which is intended specifically as a means of versioning things like
triggers and stored procedures:
http://alembic.zzzcomputing.com/en/latest/cookbook.html#replaceable-objects
Within the above Neutron patch, one thing I want to move towards is that
things like triggers and SPs would only need to be specified once, in
the migration layer, and not within the model. To achieve this, tests
that work against MySQL and Postgresql would need to ensure that the
test schema is built up using migrations, and not create_all. This is
already the case in some places and not in others. There is work
ongoing in oslo.db to provide a modernized fixture system that supports
enginefacade cleanly as well as allows for migrations to be used
efficiently (read: once per many tests) for all MySQL/Postgresql test
suites, athttps://review.openstack.org/#/c/351411/ .
As far as SQLite, I have a simple opinion with SQLite which is that
migrations, triggers, and SPs should not be anywhere near a SQLite
database. SQLite should be used strictly for simple model unit tests,
the schema is created using create_all(), and that's it. The test
fixture system accommodates this as well.
Our primary concern at this point are how to effectively test the
triggers we write against our supported database systems, and their
various deployment variations. We might be able to easily drop SQLite
support (as it's only supported for our own test suite), but should we
expect variation in support and/or actual behavior of triggers across
the MySQLs, MariaDBs, Perconas, etc, of the world that would make it
necessary to test each of them independently? If you have operational
experience working with triggers at scale: are there landmines that we
need to be aware of? What is it going to take for us to say we support
*zero* dowtime upgrades with confidence?
*zero* downtime is an extremely difficult goal. I appreciate that
people are generally nervous about making more use of relational
database features in order to help with this, however as long as the
goal includes an application that can communicate with a database that
is literally in flux as far as its schema, this is already an exotic
goal. Triggers and stored procedures are in fact very boring.
SQLAlchemy does support these features, it's just the specific trigger
and SP languages themselves are written as strings, not as elaborate
Python expressions. This should be seen as a good thing. Trigger and
SP languages are not like SQL in that they are not very declarative at
all, they are imperative. SQLAlchemy's abstraction of SQL into Python
expressions only works to the degree that the SQL itself is primarily
declarative. It would not be feasible to take on the task of producing
an imperative stored procedure / trigger language that compiles into
vendor-specific dialects.
For the use case of Openstack database migrations, I would hope that a
subset of the triggers and SPs that support live migrations would fall
into a fixed "vocabulary", each meeting various requirements, that are
of general use across projects and could be maintained in oslo.db
itself. For more elaborate cases, like "version A has the data in
three tables, version B has them in a single JSON blob", that probably
needs more specific logic.
I would encourage project teams to please loop me in on discussions
about triggers, SPs and migrations.
Steve & Dolph
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
