I'm trying to catch up this rather long and interesting discussion, sorry for somewhat late reply.
I can see aspects of 'lazy model' support in TaskFlow: - how tasks are executed and reverted - how flows are run - how engine works internally Let me address those aspects separately. == Executing and reverting tasks == I think that should be done via different interface then running a flow (or scheduling it to run), as it is completely different thing. In current TaskFlow this interface is called task executor: https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/executor.py#L57 That is actually how our WorkerBasedEngine was implemented: it's the same engine with special task executor that schedules tasks on worker instead of running task code locally. Task executors are not aware of flows by design, all they do is executing and reverting tasks. That means that task executors can be easily shared between engines if that's wanted. Current TaskExecutorBase interface uses futures (PEP 3148-like). When I proposed it, futures looked like good tool for the task at hand (see e.g. async task etherpad https://etherpad.openstack.org/p/async-taskflow-tasks) Now it may be time to reconsider that: having one future object per running task may become a scalability issue. It may be worth to use callbacks instead. It should not be too hard to refactor current engine for that. Also, as TaskExecutorBase is an internal API, there should not be any compatibility issues. Then, we can make task executor interface public and allow clients to provide their own task executors. It will be possible then for Mistral to implement its own task executor, or several, and share the executors between all the engine instances. You can call it a plan;) == Running flows == To run the flow TaskFlow client uses engine interface; also, there are few of helper functions provided for convenience: http://docs.openstack.org/developer/taskflow/engines.html#module-taskflow.engines.base http://docs.openstack.org/developer/taskflow/engines.html#creating-engines That is part of our public API, it is stable and good enough. Basically, I don't think this API needs any major change. Maybe it worth to add function or method to schedule running flow without actually waiting for flow completion (at least, it was on my top secret TODO list for quite a long time). == Engine internals == Each engine eats resources, like thread it runs on; using these resources to run one flow only is somewhat wasteful. Some work is already planned to address this situation (see e.g. https://blueprints.launchpad.net/taskflow/+spec/share-engine-thread). Also, it might be good idea to implement different 'type' of engine to support 'lazy' model, as Joshua suggests. But whatever should and will be done about it, I daresay all that work can be done without affecting API more then I described above. -- WBR, Ivan A. Melnikov ... tasks must flow ... On 02.04.2014 01:51, Dmitri Zimine wrote: > Even more responses inline :) [...] _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
