On 9/5/2018 10:03 AM, Mohammed Naser wrote:
On Wed, Sep 5, 2018 at 10:57 AM Matt Riedemann<mriede...@gmail.com> wrote:
On 9/5/2018 8:47 AM, Mohammed Naser wrote:
Could placement not do what happened for a while when the nova_api
database was created?
Can you be more specific? I'm having a brain fart here and not
remembering what you are referring to with respect to the nova_api DB.
I think there was a period in time where the nova_api database was created
where entires would try to get pulled out from the original nova database and
then checking nova_api if it doesn't exist afterwards (or vice versa). One
of the cases that this was done to deal with was for things like instance types
or flavours.
I don't know the exact details but I know that older instance types exist in
the nova db and the newer ones are sitting in nova_api. Something along
those lines?
Yeah that more about supporting online data migrations *within* nova
where new records were created in the API DB and old records would be
looked up in both the API DB and then if not found there, in the cell
(traditional nova DB). But you'd also be running the "nova-manage db
online_data_migrations" CLI to force the migration of the records from
the cell DB to the API DB.
With Placement split out of nova, we can't really do that. You could
point placement at the nova_api DB so it can pull existing records, but
it would continue to create new records in the nova_api DB rather than
the placement DB and at some point you have to make that data migration.
Maybe you were thinking something like have temporary fallback code in
placement such that if a record isn't found in the placement database,
it queries a configured nova_api database? That'd be a ton of work at
this point, and if it was something we were going to do, we should have
agreed on that in YVR several months ago, definitely pre-extraction.
I say this because I know that moving the database is a huge task for
us, considering how big it can be in certain cases for us, and it
means control plane outage too
I'm pretty sure you were in the room in YVR when we talked about how
operators were going to do the database migration and were mostly OK
with what was discussed, which was a lot will just copy and take the
downtime (I think CERN said around 10 minutes for them, but they aren't
a public cloud either), but others might do something more sophisticated
and nova shouldn't try to pick the best fit for all.
If we're provided the list of tables used by placement, we could considerably
make the downtime smaller because we don't have to pull in the other huge
tables like instances/build requests/etc
There are no instances records in the API DB, maybe you mean
instance_mappings? But yes I get the point.
What happens if things like server deletes happen while the placement service
is down?
The DELETE /allocations/{consumer_id} requests from nova to placement
will fail with some keystoneauth1 exception, but because of our old
friend @safe_connect we likely won't fail the server delete because we
squash the exception from KSA:
https://github.com/openstack/nova/blob/0f102089dd0b27c7d35f0cbba87332414032c0a4/nova/scheduler/client/report.py#L2069
However, you'd still have allocations in placement against resource
providers (compute nodes) for instances that no longer exist, which
means you're available capacity for scheduling new requests is
diminished until those bogus allocations are purged from placement,
which will take some scripting.
In other words, not good things.
I'm definitely interested in what you do plan to do for the database
migration to minimize downtime.
At this point, I'm thinking turn off placement, setup the new one, do
the migration
of the placement-specific tables (this can be a straightforward documented task
OR it would be awesome if it was a placement command (something along
the lines of `placement-manage db import_from_nova`) which would import all
the right things
You wouldn't also stop nova-api while doing this? Otherwise you're going
to get into the data/resource tracking mess described above which will
require some post-migration cleanup scripting.
The idea of having a command would be*extremely* useful for deployment tools
in automating the process and it also allows the placement team to selectively
decide what they want to onboard?
Just throwing ideas here.
--
Thanks,
Matt
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
