On 05/18/2017 02:49 PM, Sean Dague wrote:
On 05/18/2017 01:02 PM, Mike Bayer wrote:
On 05/17/2017 02:38 PM, Sean Dague wrote:
Some of the concerns/feedback has been "please describe things that are
harder by this being an abstraction", so examples are provided.
so let's go through this list:
- OpenStack services taking a more active role in managing the DBMS
mmmm, "managing" is vague to me, are we referring to the database
service itself, e.g. starting / stopping / configuring? installers
like tripleo do this now, pacemaker is standard in HA for control of
services, I think I need some background here as to what the more active
role would look like.
I will leave that one for mordred, it was his concern.
I have written a novel on this topic just now in a thread titled
"[tc] Active or passive role with our database layer"
- The ability to have zero down time upgrade for services such as
Keystone.
So "zero down time upgrades" seems to have broken into:
* "expand / contract with the code carefully dancing around the
existence of two schema concepts simultaneously", e.g. nova, neutron.
AFAIK there is no particular issue supporting multiple backends on this
because we use alembic or sqlalchemy-migrate to abstract away basic
ALTER TABLE types of feature.
Agree. But there are still issues with designing the schema upgrades
themselves to be compatible with replication streams or other online
schema update constraints.
* "expand / contract using server side triggers to reconcile the two
schema concepts", e.g. keystone. This is more difficult because there
is currently no "trigger" abstraction layer. Triggers represent more
of an imperative programming model vs. typical SQL, which is why I've
not taken on trying to build a one-size-fits-all abstraction for this in
upstream Alembic or SQLAlchemy. However, it is feasible to build a
"one-size-that-fits-openstack-online-upgrades" abstraction. I was
trying to gauge interest in helping to create this back in the
"triggers" thread, in my note at
http://lists.openstack.org/pipermail/openstack-dev/2016-August/102345.html,
which also referred to some very raw initial code examples. However, it
received strong pushback from a wide range of openstack veterans, which
led me to believe this was not a thing that was happening. Apparently
Keystone has gone ahead and used triggers anyway, however I was not
pulled into that process. But if triggers are to be "blessed" by at
least some projects, I can likely work on this problem for MySQL /
Postgresql agnosticism. If keystone is using triggers right now for
online upgrades, I would ask, are they currently working on Postgresql
as well with PG-specific triggers, or does Postgresql degrade into a
"non-online" migration scenario if you're running Keystone?
This is the triggers conversation, which while I have issues with, is
the only path forward now if you are doing keystone in a load balancer
and need to retain HA through the process.
I also have issues with this- and I continue to reject categorically the
assertion that it's the only path forward.
It's not a normal or suggested way to deal with this. There ARE
best-practice suggested ways to deal with this ... but to the point of
the other email, they require being more intimate with the HA architecture.
No one is looking at pg here. And yes, everything not mysql would just
have to take the minimal expand / contract downtime. Data services like
Keystone / Glance whose data is their REST API definitely have different
concerns than Nova dropping it's control plane for 30s to recycle code
and apply db schema tweaks.
Depending on the app, nova's control plane is just as much of a concern.
I agree- there are certainly plenty of workloads out there where it's
not - but there is an issue at hand that needs to be solved and needs to
be solved one time and then always work.
- Consistent UTF8 4 & 5 byte support in our APIs
"5 byte support" appears to refer to utf-8's ability to be...well a
total of 6 bytes. But in practice, unicode itself only needs 4 bytes
and that is as far as any database supports right now since they target
unicode (see https://en.wikipedia.org/wiki/UTF-8#Description). That's
all any database we're talking about supports at most. So...lets assume
this means four bytes.
The 5 byte statement came in via a bug to Nova, it might have been
confused, and I might have been confused in interpretting it. Lets
assume it's invalid now and move to 4 byte.
Yes.
From the perspective of database-agnosticism with regards to database
and driver support for non-ascii characters, this problem has been
solved by SQLAlchemy well before Python 3 existed when many DBAPIs would
literally crash if they received a u'' string, and the rest of them
would churn out garbage; SQLAlchemy implemented a full encode/decode
layer on top of the Python DBAPI to fix this. The situation is vastly
improved now that all DBAPIs support unicode natively.
However, on the MySQL side there is this complexity that their utf-8
support is a 3-byte only storage model, and you have to use utf8mb4 if
you want the four byte model. I'm not sure right now what projects are
specifically hitting issues related to this.
Postgresql doesn't have such a limitation. If your Postgresql server
or specific database is set up for utf-8 (which should be the case),
then you get full utf-8 character set support.
So I don't see the problem of "consistent utf8 support" having much to
do with whether or not we support Posgtresql - you of course need your
"CREATE DATABASE" to include the utf8 charset like we do on MySQL, but
that's it.
That's where we stand which means that we're doing 3 byte UTF8 on MySQL,
and 4 byte on PG. That's actually an API facing difference today. It's
work to dig out of from the MySQL side, maybe the PG one is just all
super cool and done. But it's still a consideration point.
The biggest concern for me is that we're letting API behavior be
dictated by database backend and/or database config choices. The API
should behave like the API behaves.
- The requirement that Postgresql libraries are compiled for new users
trying to just run unit tests (no equiv is true for mysql because of
the pure python driver).
I would suggest that new developers for whom the presence of things like
postgresql client libraries is a challenge (but somehow they are running
a MySQL server for their pure python driver to talk to?) don't actually
have to worry about running the tests against Postgresql, this is how
the "opportunistic" testing model in oslo.db has always worked; it only
runs for the backends that you have set up.
Also, openstack got all the way through Kilo approximately using the
native python-MySQL driver which required a compiled client library as
well as the MySQL dependencies be installed. The psycopg2 driver has a
ton of whl's up on pypi (https://pypi.python.org/pypi/psycopg2) and all
linux distros supply it as a package in any case, so an actual "compile"
should not be needed. Also, this is Openstack....it's basic existence
is a kind of this vastly enormous glue between thousands of native
(ultimately C-compiled) libraries and packages, and it runs only
on...linux. So this is just a weird point to bring up. Seems a
little red herring to me.
They aren't running a database. This is one of those areas where pypi
wheel building got a lot better since the last time I looked, and one of
the reasons we had this whole bindep system in OpenStack. Because of how
test-requirements are installed, the drivers are installed in tox
whether or not they are used, because there is no good way to late
install them in runs. Whether or not they are used is based on whether a
db is setup.
But, cool, pypi wheels are good enough that we can delete the need for
all these headers for end users, very cool.
- Consistency around case sensitivity collation defaults that lead to
strange bugs around searching/updating names in resources.
Finally, a real issue-ish thing that requires a resolution. So the
good news here is that while MySQL is defaulting to case-insensitive
collations (which I assume we like) but Postgresql has almost no support
for case-insensitive collations (unless you use this odd CITEXT
datatype), it is possible to make a case-sensitive collation style
become case-insensitive at SQL query time much more easily than it would
be to go the other way.
SQLAlchemy already has some case-insensitive operators, most notably
"ilike()", which is a case-insensitive "LIKE" that is backend agnostic.
If these search queries are just using LIKE then they only need use
ilike() from SQLAlchemy instead of like().
If we are talking about the full range of operators like ==, !=, <, >
etc., and/or if we are also concerned that developers may use like()
when they really need to use ilike(), the issue can be addressed at the
typing level as well. Using SQLAlchemy, a String datatype that
guarantees case-insensitive comparisons is straightforward to construct.
This would be a Python side replacement for the String type, and
possibly Text, Unicode, etc. as needed. It does **not** imply any
database schema migrations. The hypothetical CaseInsensitiveString
would override all the comparison operators to ensure that on the
Postgresql (or other case-sensitive) backend, both sides of the
expression are embedded within the SQL LOWER() function, so that these
comparisons act case insensitively. The trick then is to get the
downstream projects to use this type (which would be in oslo.db) in
their models, which is the usual herding cats job. But this is a pretty
solvable problem.
Sure, it's work. But that's fine. The point of that list was that there
is stuff that is work because SQLA is a leaky abstraction. Which is fine
if there are people taking that work off the table.
I would not characterize this as SQLA being a leaky abstraction.
I'd say that at some point we didn't make a decision as to what we
wanted to do with text input and how it would be stored or not stored
and how it would be searched and sorted. Case sensitive collations have
been available to us the entire time, but we never decided whether our
API was case sensitive or case insensitive. OR - we *DID* decide that
our API is case insensitive the fact that it isn't on some deployments
is a bug. I'm putting money on the 'nobody made a decision' answer.
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
