On 6 January 2016 at 12:30, Ryan Brown <[email protected]> wrote: > On 12/23/2015 04:19 AM, Dougal Matthews wrote: > >> >> >> On 22 December 2015 at 17:59, Ben Nemec <[email protected] >> <mailto:[email protected]>> wrote: >> >> Can we just do git like I've been suggesting all along? ;-) >> >> More serious discussion inline. :-) >> >> On 12/22/2015 09:36 AM, Dougal Matthews wrote: >> > Hi all, >> > >> > This topic came up in the 2015-12-15 meeting[1], and again >> briefly today. >> > After working with the code that came out of the deployment library >> > spec[2] I >> > had some concerns with how we are storing the templates. >> > >> > Simply put, when we are dealing with 100+ files from >> tripleo-heat-templates >> > how can we ensure consistency in Swift without any atomicity or >> > transactions. >> > I think this is best explained with a couple of examples. >> > >> > - When we create a new deployment plan (upload all the templates >> to swift) >> > how do we handle the case where there is an error? For >> example, if we are >> > uploading 10 files - what do we do if the 5th one fails for >> some reason? >> > There is a patch to do a manual rollback[3], but I have >> concerns about >> > doing this in Python. If Swift is completely inaccessible for >> a short >> > period the rollback wont work either. >> > >> > - When deploying to Heat, we need to download all the YAML files >> from >> > Swift. >> > This can take a couple of seconds. What happens if somebody >> starts to >> > upload a new version of the plan in the middle? We could end >> up trying to >> > deploy half old and half new files. We wouldn't have a >> consistent view of >> > the database. >> > >> > We had a few suggestions in the meeting: >> > >> > - Add a locking mechanism. I would be concerned about deadlocks or >> > having to >> > lock for the full duration of a deploy. >> >> There should be no need to lock the plan for the entire deploy. It's >> not like we're re-reading the templates at the end of the deploy >> today. >> It's a one-shot read and then the plan could be unlocked, at least >> as >> far as I know. >> >> >> Good point. That would be holding the lock for longer than we need. >> >> >> The only option where we wouldn't need locking at all is the >> read-copy-update model Clint mentions, which might be a valid option >> as >> well. Whatever we do, there are going to be concurrency issues >> though. >> For example, what happens if two users try to make updates to the >> plan >> at the same time? If you don't either merge the changes or disallow >> one >> of them completely then one user's changes might be lost. >> >> TBH, this is further convincing me that we should just make this git >> backed and let git handle the merging and conflict resolution (never >> mind the fact that it gets us a well-understood version control system >> for "free"). For updates that don't conflict with other changes, git >> can merge them automatically, but for merge conflicts you just return >> a >> rebase error to the user and make them resolve it. I have a feeling >> this is the behavior we'll converge on eventually anyway, and rather >> than reimplement git, let's just use the real thing. >> >> >> I'd be curious to hear more how you would go about doing this with git. >> I've >> never automated git to this level, so I am concerned about what issues we >> might hit. >> >> Do you happen to know of any good sources that I can find out more? >> > > One source of inspiration could be the way Openshift handles things. You > get a git URL to push your $whatever to, and they have git hooks that > deploy when you push. This would be perfect for merge conflict situations. > Imagine two cases. >
Interesting. I hadn't even considered that we could act as a git remote. That would be pretty neat. It would mean that we are very tied to git, which isn't necessarily a bad thing but it is something to consider. Normal case: > 1. User does stuff in web interface or over API > 2. The API adds templates and such to git repo, then deploys as needed > 3. User is happy! > > Merge conflict case: > 1. User does stuff in web interface or over API > 2. The API tries, but there's a merge conflict. It hands back a 409 and a > git URL > 3. User clones the repo and resolves conflict > 4. User pushes repo back up > 5. API deploys stuff > 6. User is happy! > > > > /2 cents >> >> > - Store the files in a tarball (so we only deal with one file). I >> think we >> > could still hit issues with multiple operations unless we >> guarantee that >> > only one instance of the API is ever running. >> > >> > I think these could potentially be made to work, but at this point >> are we >> > using the best tool for the job? >> > >> > For alternatives, I see a can think of a couple of options: >> > >> > - Use a database, like we did for Tuskar and most OpenStack API's >> do. >> >> I kind of like this, in particular because it would allow us to handle >> migrations between versions the same way as other OpenStack services. >> >> >> Yeah, I feel like this is the "safe" option. It's a well trodden and >> tested path. >> >> I'm not entirely sure how it maps to our template configuration method >> though. Storing a bunch of template blobs in the database feels a >> little square peg, round hole to me, and might undo a lot of the >> benefits of using the database in the first place. >> >> >> I don't follow this point. In the way we access Swift, everything is >> essentially >> a blob of text also. >> >> Now, on the other hand, having a database to store basic data like >> metadata on plans and such might be a useful thing regardless of where >> we store the templates themselves. We could also use it for locking >> just fine IMHO - TripleO isn't a tool targeted to have cloud-scale >> number of users. It's a deployer tool with a relatively limited >> number >> of users per installation, so the scaling of a locking mechanism >> isn't a >> problem I necessarily think we need to solve. We have way bigger >> scaling issues than that to tackle before I think we would hit the >> limit >> of a database-locking scheme. >> >> > - Invest time in building something on Swift. >> > - Glance was transitioning to be a Artifact store. I don't know the >> > status of >> > this or if it would meet out needs. >> > >> > Any input, ideas or suggestions would be great! >> > >> > Thanks, >> > Dougal >> > >> > >> > [1]: >> > >> http://eavesdrop.openstack.org/meetings/tripleo/2015/tripleo.2015-12-15-14.03.log.html#l-89 >> > [2]: >> > >> https://specs.openstack.org/openstack/tripleo-specs/specs/mitaka/tripleo-overcloud-deployment-library.html >> > [3]:https://review.openstack.org/#/c/257481/ >> > >> > >> > >> >> __________________________________________________________________________ >> > OpenStack Development Mailing List (not for usage questions) >> > Unsubscribe: >> [email protected]?subject:unsubscribe >> <http://[email protected]?subject:unsubscribe >> > >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> >> >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> [email protected]?subject:unsubscribe >> <http://[email protected]?subject:unsubscribe >> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> [email protected]?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > -- > Ryan Brown / Senior Software Engineer, Openstack / Red Hat, Inc. > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
