For persistenting the graph there is also: https://github.com/LogicalDash/gorm#gorm
The above sounds interesting... 'Object relational mapper for graphs with in-built revision control.' It then hooks in to the https://networkx.github.io/ library which provides a ton of useful graph functionality. As you guys get more and more into doing graph manipulation I am betting that library (networkx) will become more and more useful... -Josh On Oct 23, 2014, at 11:10 AM, Zane Bitter <zbit...@redhat.com> wrote: > Hi folks, > I've been looking at the convergence stuff, and become a bit concerned that > we're more or less flying blind (or at least I have been) in trying to figure > out the design, and also that some of the first implementation efforts seem > to be around the stuff that is _most_ expensive to change (e.g. database > schemata). > > What we really want is to experiment on stuff that is cheap to change with a > view to figuring out the big picture without having to iterate on the > expensive stuff. To that end, I started last week to write a little prototype > system to demonstrate the concepts of convergence. (Note that none of this > code is intended to end up in Heat!) You can find the code here: > > https://github.com/zaneb/heat-convergence-prototype > > Note that this is a *very* early prototype. At the moment it can create > resources, and not much else. I plan to continue working on it to implement > updates and so forth. My hope is that we can develop a test framework and > scenarios around this that can eventually be transplanted into Heat's > functional tests. So the prototype code is throwaway, but the tests we might > write against it in future should be useful. > > I'd like to encourage anyone who needs to figure out any part of the design > of convergence to fork the repo and try out some alternatives - it should be > very lightweight to do so. I will also entertain pull requests (though I see > my branch primarily as a vehicle for my own learning at this early stage, so > if you want to go in a different direction it may be best to do so on your > own branch), and the issue tracker is enabled if there is something you want > to track. > > I have learned a bunch of stuff already: > > * The proposed spec for persisting the dependency graph > (https://review.openstack.org/#/c/123749/1) is really well done. Kudos to > Anant and the other folks who had input to it. I have left comments based on > what I learned so far from trying it out. > > > * We should isolate the problem of merging two branches of execution (i.e. > knowing when to trigger a check on one resource that depends on multiple > others). Either in a library (like taskflow) or just a separate database > table (like my current prototype). Baking it into the orchestration > algorithms (e.g. by marking nodes in the dependency graph) would be a > colossal mistake IMHO. > > > * Our overarching plan is backwards. > > There are two quite separable parts to this architecture - the worker and the > observer. Up until now, we have been assuming that implementing the observer > would be the first step. Originally we thought that this would give us the > best incremental benefits. At the mid-cycle meetup we came to the conclusion > that there were actually no real incremental benefits to be had until > everything was close to completion. I am now of the opinion that we had it > exactly backwards - the observer implementation should come last. That will > allow us to deliver incremental benefits from the observer sooner. > > The problem with the observer is that it requires new plugins. (That sucks > BTW, because a lot of the value of Heat is in having all of these tested, > working plugins. I'd love it if we could take the opportunity to design a > plugin framework such that plugins would require much less custom code, but > it looks like a really hard job.) Basically this means that convergence would > be stalled until we could rewrite all the plugins. I think it's much better > to implement a first stage that can work with existing plugins *or* the new > ones we'll eventually have with the observer. That allows us to get some > benefits soon and further incremental benefits as we convert plugins one at a > time. It should also mean a transition period (possibly with a performance > penalty) for existing plugin authors, and for things like HARestarter (can we > please please deprecate it now?). > > So the two phases I'm proposing are: > 1. (Workers) Distribute tasks for individual resources among workers; > implement update-during-update (no more locking). > 2. (Observers) Compare against real-world values instead of template values > to determine when updates are needed. Make use of notifications and such. > > I believe it's quite realistic to aim to get #1 done for Kilo. There could > also be a phase 1.5, where we use the existing stack-check mechanism to > detect the most egregious divergences between template and reality (e.g. > whole resource is missing should be easy-ish). I think this means that we > could have a feasible Autoscaling API for Kilo if folks step up to work on it > - and in any case now is the time to start on that to avoid it being delayed > more than it needs to be based purely on the availability of underlying > features. That's why I proposed a session on Autoscaling for the design > summit. > > > * This thing is going to _hammer_ the database > > The advantage is that we'll be able to spread the access across an arbitrary > number of workers, but it's still going to be brutal because there's only one > database. We'll need to work hard to keep DB access to the minimum we can. > > > That's it for now. I'm going to continue hacking on this, though don't expect > much progress before the Summit. I'd be happy to hear any feedback that > anybody has, and I hope this might prove a useful platform for other folks to > experiment with too. > > cheers, > Zane. > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
