I left a very large comment on Gerrit but I'll add it here for better visibility.
FWIW, I think the problem is legit and needs to be addressed. I'm gonna change the title and the subject to make it clearer but I also think that the solution isn't simple at first and requires some design discussion, hence the Wishlist status. Now, the comment I wrote explaining my -1 (you can find it here https://review.opendev.org/c/openstack/nova/+/864760/comment/b2b03637_f15d6dd2/ ) ================= > Just because you say so? =) > Can you provide a more technical explanation on why not? I mean, why would that be wrong? Or, what alternative would be better, and why? Sorry, that's kind of a non-documented design consensus (or a tribal knowledge if you prefer) We, as the Nova community, want to keep the RequestSpec.availability_zone record as an immutable object, that is only set when creating the RequestSpec, so then we know whether the user wanted to pin the instance to a specific AZ or not. > What is your proposal? We see the following two different alternatives so far. [...] Maybe you haven't seen my proposal before, but I was talking of https://review.opendev.org/c/openstack/nova/+/469675/12/nova/compute/api.py#1173 that was merged. See again my comment https://review.opendev.org/c/openstack/nova/+/864760/comments/4a302ce3_9805e7c6 TBC, lemme explain the problem and what we need to fix : if an user creates an instance with an image and asking to create a volume on that image, then we need to modify the AZ for the related Request if and only if cross_az_attach=False Now, let's discuss the implementation : 1/ we know that volumes are created way later in the instance boot by the compute service, but we do pass the information of the instance.az to Cinder to tell it to create a volume within that AZ if cross_az_attach=False : https://github.com/openstack/nova/blob/b3fdd7ccf01bafb68e37a457f703b79119dbfa86/nova/virt/block_device.py#L427 https://github.com/openstack/nova/blob/b3fdd7ccf01bafb68e37a457f703b79119dbfa86/nova/virt/block_device.py#L53-L78 2/ unfortunately,instance.availability_zone is only trustworthy if the instance is pinned to an AZ 3/ we know that at the API level, we're able to know whether we will create a volume based on an image since we have the BDMs and we do check them : https://github.com/openstack/nova/blob/b3fdd7ccf01bafb68e37a457f703b79119dbfa86/nova/compute/api.py#L1460 https://github.com/openstack/nova/blob/b3fdd7ccf01bafb68e37a457f703b79119dbfa86/nova/compute/api.py#L1866 https://github.com/openstack/nova/blob/b3fdd7ccf01bafb68e37a457f703b79119dbfa86/nova/compute/api.py#L1960-L1965C43 4/ Accordingly, we are able to follow the same logic than in https://github.com/openstack/nova/blob/b3fdd7ccf01bafb68e37a457f703b79119dbfa86/nova/compute/api.py#L1396-L1397 by checking the BDMs and see whether we gonna create a volume. If so, we SHALL pin the AZ exactly like https://github.com/openstack/nova/blob/b3fdd7ccf01bafb68e37a457f703b79119dbfa86/nova/compute/api.py#L1264 Unfortunately, since the user didn't specify an AZ, Nova doesn' know which AZ to pin the instance to. Consequently, we have multiple options : 1/ we could return an exception to the user if he didn't pinned the instance. That said, I really don't like this UX since the user doesn't know whether cross_az_attach is False or not 2/ we could document the fact that cross_az_attach only works with pre-created volumes. 3/ we could pre-create the volume way earlier at the API level and get its AZ. 4/ we could augment the RequestSpec to have a field saying 'pinned' or something else that the scheduler would honor on a move operation even if RequestSpec.az is None As you see, all those options need to be correctly discussed, so IMHO I'd prefer you to draft a spec so the nova community could address those points so we could find an approved design solution. HTH. ** Changed in: nova Status: Invalid => Confirmed ** Changed in: nova Importance: Undecided => Wishlist ** Summary changed: - 'openstack server resize --flavor' should not migrate VMs to another AZ + cross_az_attach=False doesn't honor BDMs with source=image and dest=volume -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2018318 Title: cross_az_attach=False doesn't honor BDMs with source=image and dest=volume Status in OpenStack Compute (nova): Confirmed Bug description: The config flag cross_az_attach allows an instance to be pinned to the related volume AZ if the value of that config option is set to False. We fixed the case of a volume-backed instance by https://review.opendev.org/c/openstack/nova/+/469675/ if the volume was created before the instance but we haven't yet resolved the case of an BFV-instance created from an image (the BDM shortcut that allows a late creation of a volume by the compute service). Since the volume is created based on the current instance.AZ, it does respect the current AZ, but since the instance isn't pinned to that AZ, it can move from an AZ to another while the volume will continue to exist in the original AZ. As a consequence, the problem is only seen after a move operation but the problem exists since the instance creation. === ORIGINAL BUG REPORT BELOW === Before I start, let me describe the agents involved in the process migration and/or resize flow of OpenStack (in this case, Nova component). These are the mapping and interpretation I created while troubleshooting the reported problem. - Nova-API: the agent responsible for receiving the HTTP requests (create/resize/migrate) from the OpenStack end-user. It does some basic validation, and then sends a message with the requested command via RPC call to other agents. - Nova-conductor: the agent responsible to "conduct/guide" the workflow. Nova-conductor will read the commands from the RPC queue and then process the request from Nova-API. It does some extra validation, and for every command (create/resize/migrate), it asks for the scheduler to define the target host for the operation (if the target host was not defined by the user). - Nova-scheduler: the agent responsible to "schedule" VMs on hosts. It defines where a VM must reside. It receives the "select host request", and processes the algorithms to determine where the VM can be allocated. Before applying the scheduling algorithms, it calls/queries the Placement system to get the possible hosts where VMs might be allocated. I mean, hosts that fit the requested parameters, such as being in a given Cell, availability zone (AZ), having available/free computing resources to support the VM. The call from Nova-scheduler to Placement is an HTTP request. - Placement: behaves as an inventory system. It tracks where resources are allocated, their characteristics, and providers (hosts/storage/network system) where resources are (can be) allocated. It also has some functions to return the possible hosts where a "request spec" can be fulfilled. - Nova: the agent responsible to execute/process the commands and implement actions in the hypervisor. Then, we have the following workflow from the different processes. - migrate: Nova API ->(via RPC call -- nova.conductor.manager.ComputeTaskManager.live_migrate_instance) Nova Conductor (loads request spec) -> (via RPC call) Nova scheduler -> (via HTTP) Placement -> (after the placement return) Nova scheduler executes the filtering of the hosts, based on active filters. - > (return for the other processes in conductor) -> (via RPC call) Nova to execute the migration. - resize: Nova API ->(via RPC call -- nova.conductor.manager.ComputeTaskManager.migrate_server -- _cold_migrate) Nova Conductor (loads request spec) -> (via RPC call) Nova scheduler -> (via HTTP) Placement -> (after the placement return) nova scheduler executes the filtering of the hosts, based on active filters - > (return for the other processes), in Nova conductor -> (RPC call) Nova to execute the cold migration and start the VM again with the new computing resource definition As a side note, this mapping also explains why the "resize" was not executing the CPU compatibility check that the "migration" is executing (this is something else that I was checking, but it is worth mentioning here). The resize is basically a cold migration to a new host, where a new flavor (definition of the VM) is applied; thus, it does not need to evaluate CPU feature set compatibility. The problem we are reporting happens with both "migrate" and "resize" operations. Therefore, I had to add some logs to see what was going on there (that whole process is/was "logless"). The issue happens because Placement always returns all hosts of the environment for a given VM being migrated (resize is a migration process); this only happens if the VM is deployed without defining its availability zone in the request spec. To be more precise, Nova-conductor in `nova.conductor.tasks.live_migrate.LiveMigrationTask._get_request_spec_for_select_destinations` (https://github.com/openstack/nova/blob/3d83bb3356e10355437851919e161f258cebf761/nova/conductor/tasks/live_migrate.py#L460) always uses the original request specification, used to deploy the VM, to find a new host to migrate it to. Therefore, if the VM is deployed to a specific AZ, it will always send this AZ to Placement (because the AZ is in the request spec), and Placement will filter out hosts that are not from that AZ. However, if the VM is deployed without defining the AZ, Nova will select a host (from an AZ) to deploy it (the VM), and when migrating the VM, Nova is not trying to find another host in the same AZ where the VM is already running. It is always behaving as a new deployment process to select the host. That raised a question. How is it possible that the create (deploy VM) process works? It works because of the parameter "cross_az_attach" configured in Nova. As we can see in https://github.com/openstack/nova/blob/3d83bb3356e10355437851919e161f258cebf761/nova/virt/block_device.py#L53, if this parameter is False, when creating a volume, Nova is going to use the AZ where the VM was scheduled to create the volume in Cinder. Everything works because the host selection process is executed before the volume is created in Cinder. After discovering all that, we were under the impression that OpenStack was designed to have (require) different Cells to implement multiple AZs. Therefore, we assumed that the problem was caused due to this code/line (https://github.com/openstack/nova/blob/3d83bb3356e10355437851919e161f258cebf761/nova/conductor/tasks/live_migrate.py#L495). Whenever a request is made to Nova-scheduler, Nova conductor always sends the current Cell where the VM resides to Nova Placement. Therefore, if we had multiple AZs, each one of them with different Cell configurations, we would never have had this situation; that is why we were thinking that the problem might be a setup one. However, while discussing, and after checking the documentation (https://docs.openstack.org/nova/latest/admin/availability-zones.html) that describes the use of AZs, we concluded that there is an issue with the code. It should be possible to have multiple AZs sharing the same Cell. We conclude that similar to what happens when "cross_az_attach" is False and we deploy a VM, Nova is going to allocate the Cinder volume in a specific AZ, then when executing migrations this parameter "cross_az_attach" should be evaluated, and the current AZ of the VM should be added in the request spec to Placement to list the possible hosts where the VM can be moved to. We also discussed if the Placement should be the one doing this check before returning the possible hosts to migrate the VM to. However, this does not seem to be in the Placement context/goal/design. Therefore, the place where we need a patch/fix is in Nova. Furthermore, the solution proposed (https://review.opendev.org/c/openstack/nova/+/469675/12/nova/compute/api.py#1173) is only addressing the cases when the VM is created based on volumes, and then it sets the AZ of the volumes in the request spec of the VM (even though the user is not setting that in the request spec of the VM). That is why everything works for the setups where cross_az_attach=False. However, if we create a VM based on an image, and then it (Nova) creates a new volume in Cinder, the AZ is not set in the request spec, but it (the request spec) is used to execute the first call to placement to select the hosts, as we described above here. Following the same process that is used with Nova cells, we propose the solution for this situation at https://review.opendev.org/c/openstack/nova/+/864760. Any other comments and reviews are welcome! To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2018318/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
