On 05/31/2018 01:09 PM, Dan Smith wrote:
My feeling is that we should not attempt to "migrate" any allocations
or inventories between root or child providers within a compute node,
period.

While I agree this is the simplest approach, it does put a lot of
responsibility on the operators to do work to sidestep this issue, which
might not even apply to them (and knowing if it does might be
difficult).

Perhaps, yes. Though the process I described is certainly not foreign to operators. It is a safe and well-practiced approach.

The virt drivers should simply error out of update_provider_tree() if
there are ANY existing VMs on the host AND the virt driver wishes to
begin tracking resources with nested providers.

The upgrade operation should look like this:

1) Upgrade placement
2) Upgrade nova-scheduler
3) start loop on compute nodes. for each compute node:
  3a) disable nova-compute service on node (to take it out of scheduling)
  3b) evacuate all existing VMs off of node

You mean s/evacuate/cold migrate/ of course... :)

I meant evacuate as in `nova host-evacuate-live` with a fall back to `nova host-servers-migrate` if live migration isn't possible).

  3c) upgrade compute node (on restart, the compute node will see no
      VMs running on the node and will construct the provider tree inside
      update_provider_tree() with an appropriate set of child providers
      and inventories on those child providers)
  3d) enable nova-compute service on node

Which is virtually identical to the "normal" upgrade process whenever
there are significant changes to the compute node -- such as upgrading
libvirt or the kernel.

Not necessarily. It's totally legit (and I expect quite common) to just
reboot the host to take kernel changes, bringing back all the instances
that were there when it resumes.

So, you're saying the normal process is to try upgrading the Linux kernel and associated low-level libs, wait the requisite amount of time that takes (can be a long time) and just hope that everything comes back OK? That doesn't sound like any upgrade I've ever seen. All upgrade procedures I have seen attempt to get the workloads off of the compute host before trying anything major (and upgrading a linux kernel or low-level lib like libvirt is a major thing IMHO).

> The "normal" case of moving things
around slide-puzzle-style applies to live migration (which isn't an option here).
Sorry, I was saying that for all the lovely resources that have been bolted on to Nova in the past 5 years (CPU pinning, NUMA topologies, PCI passthrough, SR-IOV PF/VFs, vGPUs, etc), that if the workload uses *those* resources, then live migration won't work and that the admin would need to fall back to nova host-servers-migrate. I wasn't saying that live migration for all workloads/instances would not be a possibility.

I think people that can take downtime for the instances would rather
not have to move things around for no reason if the instance has to
get shut off anyway.

Maybe. Not sure. But my line of thinking is stick to a single, already known procedure since that is safe and well-practiced.

Code that we don't have to write means code that doesn't have new bugs that we'll have to track down and fix.

I'm also thinking that we'd be tracking down and fixing those bugs while trying to put out a fire that was caused by trying to auto-heal everything at once on nova-compute startup and resulting in broken state and an inability of the nova-compute service to start again, essentially trapping instances on the failed host. ;)

Nested resource tracking is another such significant change and should
be dealt with in a similar way, IMHO.

This basically says that for anyone to move to rocky, they will have to
cold migrate every single instance in order to do that upgrade right?

No, sorry if I wasn't clear. They can live-migrate the instances off of the to-be-upgraded compute host. They would only need to cold-migrate instances that use the aforementioned non-movable resources.

I kinda think we need to either:

1. Make everything perform the pivot on compute node start (which can be
    re-used by a CLI tool for the offline case)
>
2. Make everything default to non-nested inventory at first, and provide
    a way to migrate a compute node and its instances one at a time (in
    place) to roll through.

I would vote for Option #2 if it comes down to it.

If we are going to go through the hassle of writing a bunch of transformation code in order to keep operator action as low as possible, I would prefer to consolidate all of this code into the nova-manage (or nova-status) tool and put some sort of attribute/marker on each compute node record to indicate whether a "heal" operation has occurred for that compute node.

Kinda like what Matt's been playing with for the heal_allocations stuff.

At least in that case, we'd have all the transform/heal code in a single place and we wouldn't need to have much, if any, code in the compute manager, resource tracker or "scheduler" report client.

Someone (maybe Gibi?) on this thread had mentioned having the virt driver (in update_provider_tree) do the whole set reserved = total thing when first attempting to create the child providers. That would work to prevent the scheduler from attempting to place workloads on those child providers, but we would still need some marker on the compute node to indicate to the nova-manage heal_nested_providers (or whatever) command that the compute node has had its provider tree validated/healed, right?

We can also document "or do the cold-migration slide puzzle thing" as an
alternative for people that feel that's more reasonable.

I just think that forcing people to take down their data plane to work
around our own data model is kinda evil and something we should be
avoiding at this level of project maturity.

The use of the word "evil" is a little, well, brutal, to describe something I'm proposing that would just be more work for operators but (again, IMHO) be the safest proven method for solving this problem. :)

Best,
-jay

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to