So just attempting to read through this thread, I think I hear: Problems:
1. multi-attach breaks the assumption that made detach work 2. live-migrate, already breaks with some drivers, due to not fully understanding the side affects of all API calls. 3. evacuate and shelve issues also related Solution ideas: 1. New export/target for every volume connection * pro: simple * con: that doesn't work for all drivers (?) 2. Nova works out when to disconnect volume on host * pro: no cinder API changes (i.e. no upgrade issue) * con: adds complexity in Nova * con: requires all nodes to run fixed code before multi-attach is safe * con: doesn't help with the live-migrate and evacuate issues anyways? 3. Give Cinder all the info, so it knows what has to happen * pro: seems to give cinder the info to stop API users doing bad things * pro: more robust API particularly useful with multiple nova, and with baremetal, etc * con: Need cinder micro-versions to do this API change and work across upgrade So from where I am sat: 1: doesn't work for everyone 2: doesn't fix all the problems we need to fix 3: will take a long time If so, it feels like we need solution 3, regardless, to solve wider issues. We only need solution 2, if solution 3 will block multi-attach for too long. Am I missing something in that summary? Thanks, johnthetubaguy On 12 February 2016 at 20:26, Ildikó Váncsa <ildiko.van...@ericsson.com> wrote: > Hi Walt, > > Thanks for describing the bigger picture. > > In my opinion when we will have microversion support available in Cinder that > will give us a bit of a freedom and also possibility to handle these > difficulties. > > Regarding terminate_connection we will have issues with live_migration as it > is today. We need to figure out what information would be best to feed back > to Cinder from Nova, so we should figure out what API we would need after we > are able to introduce it in a safe way. I still see benefit in storing the > connection_info for the attachments. > > Also I think the multiattach support should be disable for the problematic > drivers like lvm, until we don't have a solution for proper detach on the > whole call chain. > > Best Regards, > Ildikó > >> -----Original Message----- >> From: Walter A. Boring IV [mailto:walter.bor...@hpe.com] >> Sent: February 11, 2016 18:31 >> To: openstack-dev@lists.openstack.org >> Subject: Re: [openstack-dev] [Nova][Cinder] Multi-attach, determining when >> to call os-brick's connector.disconnect_volume >> >> There seems to be a few discussions going on here wrt to detaches. One >> is what to do on the Nova side with calling os-brick's disconnect_volume, >> and also when to or not to call Cinder's >> terminate_connection and detach. >> >> My original post was simply to discuss a mechanism to try and figure out the >> first problem. When should nova call brick to remove the >> local volume, prior to calling Cinder to do something. >> >> Nova needs to know if it's safe to call disconnect_volume or not. Cinder >> already tracks each attachment, and it can return the >> connection_info >> for each attachment with a call to initialize_connection. If 2 of >> those connection_info dicts are the same, it's a shared volume/target. >> Don't call disconnect_volume if there are any more of those left. >> >> On the Cinder side of things, if terminate_connection, detach is called, the >> volume manager can find the list of attachments for a >> volume, and compare that to the attachments on a host. The problem is, >> Cinder doesn't track the host along with the instance_uuid in >> the attachments table. I plan on allowing that as an API change after >> microversions lands, so we know how many times a volume is >> attached/used on a particular host. The driver can decide what to do with >> it at >> terminate_connection, detach time. This helps account for >> the differences in each of the Cinder backends, which we will never get all >> aligned to the same model. Each array/backend handles >> attachments different and only the driver knows if it's safe to remove the >> target or not, depending on how many attachments/usages >> it has >> on the host itself. This is the same thing as a reference counter, >> which we don't need, because we have the count in the attachments table, >> once we allow setting the host and the instance_uuid at >> the same time. >> >> Walt >> > On Tue, Feb 09, 2016 at 11:49:33AM -0800, Walter A. Boring IV wrote: >> >> Hey folks, >> >> One of the challenges we have faced with the ability to attach a >> >> single volume to multiple instances, is how to correctly detach that >> >> volume. The issue is a bit complex, but I'll try and explain the >> >> problem, and then describe one approach to solving one part of the detach >> >> puzzle. >> >> >> >> Problem: >> >> When a volume is attached to multiple instances on the same host. >> >> There are 2 scenarios here. >> >> >> >> 1) Some Cinder drivers export a new target for every attachment on >> >> a compute host. This means that you will get a new unique volume >> >> path on a host, which is then handed off to the VM instance. >> >> >> >> 2) Other Cinder drivers export a single target for all instances >> >> on a compute host. This means that every instance on a single host, >> >> will reuse the same host volume path. >> > >> > This problem isn't actually new. It is a problem we already have in >> > Nova even with single attachments per volume. eg, with NFS and SMBFS >> > there is a single mount setup on the host, which can serve up multiple >> > volumes. >> > We have to avoid unmounting that until no VM is using any volume >> > provided by that mount point. Except we pretend the problem doesn't >> > exist and just try to unmount every single time a VM stops, and rely >> > on the kernel failing umout() with EBUSY. Except this has a race >> > condition if one VM is stopping right as another VM is starting >> > >> > There is a patch up to try to solve this for SMBFS: >> > >> > https://review.openstack.org/#/c/187619/ >> > >> > but I don't really much like it, because it only solves it for one >> > driver. >> > >> > I think we need a general solution that solves the problem for all >> > cases, including multi-attach. >> > >> > AFAICT, the only real answer here is to have nova record more info >> > about volume attachments, so it can reliably decide when it is safe to >> > release a connection on the host. >> > >> > >> >> Proposed solution: >> >> Nova needs to determine if the volume that's being detached is a >> >> shared or non shared volume. Here is one way to determine that. >> >> >> >> Every Cinder volume has a list of it's attachments. In those >> >> attachments it contains the instance_uuid that the volume is attached >> >> to. I presume Nova can find which of the volume attachments are on >> >> the same host. Then Nova can call Cinder's initialize_connection for >> >> each of those attachments to get the target's connection_info >> >> dictionary. This connection_info dictionary describes how to connect >> >> to the target on the cinder backend. If the target is shared, then >> >> each of the connection_info dicts for each attachment on that host >> >> will be identical. Then Nova would know that it's a shared target, >> >> and then only call os-brick's disconnect_volume, if it's the last >> >> attachment on that host. I think at most 2 calls to cinder's >> >> initialize_connection would suffice to determine if the volume is a >> >> shared target. This would only need to be done if the volume is >> >> multi-attach capable and if there are more than 1 attachments on the same >> >> host, where the detach is happening. >> > As above, we need to solve this more generally than just multi-attach, >> > even single-attach is flawed today. >> > >> > Regards, >> > Daniel >> >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
