Your example sounds a lot like what taskflow is build for doing. https://github.com/stackforge/taskflow/blob/master/taskflow/examples/calculate_in_parallel.py is a decent example.
In that one, tasks are created and input/output dependencies are specified (provides, rebind, and the execute function arguments itself). This is combined into the taskflow concept of a flow, one of those flows types is a dependency graph. Using a parallel engine (similar in concept to a heat engine) we can run all non-dependent tasks in parallel. An example that I just created that shows this (and shows it running) that closer matches your example. Program (this will work against the current taskflow codebase): http://paste.openstack.org/show/48156/ Output @ http://paste.openstack.org/show/48157/ -Josh From: Lakshminaraya Renganarayana <lren...@us.ibm.com<mailto:lren...@us.ibm.com>> Reply-To: OpenStack Development Mailing List <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> Date: Wednesday, October 9, 2013 11:31 AM To: OpenStack Development Mailing List <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows Steven Hardy <sha...@redhat.com<mailto:sha...@redhat.com>> wrote on 10/09/2013 05:24:38 AM: > > So as has already been mentioned, Heat defines an internal workflow, based > on the declarative model defined in the template. > > The model should define dependencies, and Heat should convert those > dependencies into a workflow internally. IMO if the user also needs to > describe a workflow explicitly in the template, then we've probably failed > to provide the right template interfaces for describing depenendencies. I agree with Steven here, models should define the dependencies and Heat should realize/enforce them. An important design issue is granularity at which dependencies are defined and enforced. I am aware of the wait-condition and signal constructs in Heat, but I find them a bit low-level as they are prone to the classic dead-lock and race condition problems. I would like to have higher level constructs that support finer-granularity dependences which are needed for software orchestration. Reading through the various disucssion on this topic in this mailing list, I see that many would like to have such higher level constructs for coordination. In our experience with software orchestration using our own DSL and also with some extensions to Heat, we found that the granularity of VMs or Resources to be too coarse for defining dependencies for software orchestration. For example, consider a two VM app, with VMs vmA, vmB, and a set of software components (ai's and bi's) to be installed on them: vmA = base-vmA + a1 + a2 + a3 vmB = base-vmB + b1 + b2 + b3 let us say that software component b1 of vmB, requires a config value produced by software component a1 of vmA. How to declaratively model this dependence? Clearly, modeling a dependence between just base-vmA and base-vmB is not enough. However, defining a dependence between the whole of vmA and vmB is too coarse. It would be ideal to be able to define a dependence at the granularity of software components, i.e., vmB.b1 depends on vmA.a1. Of course, it would also be good to capture what value is passed between vmB.b1 and vmA.a1, so that the communication can be facilitated by the orchestration engine. We found that such finer granular modeling of the dependencies provides two valuable benefits: 1. Faster total (resources + software setup) deployment time. For the example described above, a coarse-granularity dependence enforcer would start the deployment of base-vmB after vmA + a1 + a2 + a3 is completed, but a fine-granularity dependence enforcer would start base-vmA and base-vmB concurrently, and then suspend the execution of vmB.b1 until vmA.a1 is complete and then let the rest of deployment proceed concurrently, resulting in a faster completion. 2. More flexible dependencies. For example, mutual dependencies between resources, which can be satisfied when orchestrated at a finer granularity. Using the example described above, fine-granularity would allow vmB.b1 depends_on vmA.a1 and also vmA.a3 depends_on vmB.b2, but coarse-granularity model would flag this as a cyclic dependence. There are two aspects that needs support: 1. Heat/HOT template level constructs to support declarative expression of such fine-granularity dependencies and the values communicated / passed for the dependence. 2. Support from Heat engine / analyzer in supporting the runtime ordering, coordination between resources, and also the communication of the values. What are your thoughts?
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
