On 22 December 2015 at 16:56, Clint Byrum <[email protected]> wrote: > Excerpts from Dougal Matthews's message of 2015-12-22 07:36:02 -0800: > > Hi all, > > > > This topic came up in the 2015-12-15 meeting[1], and again briefly today. > > After working with the code that came out of the deployment library > spec[2] > > I > > had some concerns with how we are storing the templates. > > > > Simply put, when we are dealing with 100+ files from > tripleo-heat-templates > > how can we ensure consistency in Swift without any atomicity or > > transactions. > > I think this is best explained with a couple of examples. > > > > - When we create a new deployment plan (upload all the templates to > swift) > > how do we handle the case where there is an error? For example, if we > are > > uploading 10 files - what do we do if the 5th one fails for some > reason? > > There is a patch to do a manual rollback[3], but I have concerns about > > doing this in Python. If Swift is completely inaccessible for a short > > period the rollback wont work either. > > > > You could create a unique swift container, upload things to that, and > then update a pointer in a well-known location to point at that container > for the new plan only after you've verified it is available. This is a > primitive form of Read-copy-update. > > > - When deploying to Heat, we need to download all the YAML files from > > Swift. > > This can take a couple of seconds. What happens if somebody starts to > > upload a new version of the plan in the middle? We could end up > trying to > > deploy half old and half new files. We wouldn't have a consistent > view of > > the database. > > > > Perhaps you should land a change in Heat to allow templates to be directly > downloaded by the engine from swift without needing to be uploaded. In > the past allowing URLs to be downloaded unfettered was disabled because > we don't want Heat to be a DoS engine, but swift would be in-cloud and > could be restricted to the stack owner. > > > We had a few suggestions in the meeting: > > > > - Add a locking mechanism. I would be concerned about deadlocks or > having > > to > > lock for the full duration of a deploy. > > Deadlocks only happen when competing interests lock _two_ things in > different order. Have one thing to lock, and you don't have to worry > about this. >
Good point. Sorry, I am getting my terminology confused. I can't think what the word is, but I was worried about stale locks getting lost and locking everything up. Where would we store the lock? On the machine running the API maybe? > - Store the files in a tarball (so we only deal with one file). I think > we > > could still hit issues with multiple operations unless we guarantee > that > > only one instance of the API is ever running. > > > > I think these could potentially be made to work, but at this point are we > > using the best tool for the job? > > > > For alternatives, I see a can think of a couple of options: > > > > - Use a database, like we did for Tuskar and most OpenStack API's do. > > It's worth noting that many OpenStack API's/daemons are using the > database + MQ to provide consistency in a distributed fashion, and many > have identified that this doesn't scale particularly well, and looking > at tooz to help bring in DLM's. In fact, a spec recently landed around > this: > > > http://specs.openstack.org/openstack/openstack-specs/specs/chronicles-of-a-dlm.html > > So if you are only using the DB for consistency, you might want to just > use tooz+swift. > Interesting! I hadn't come across this and I'll look into it. Thanks. > > > - Invest time in building something on Swift. > > - Glance was transitioning to be a Artifact store. I don't know the > status > > of > > this or if it would meet out needs. > > I'd say the artifact store is more about exposing options to users, and > less about providing primitives for an API. > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
