Thank Ivan, This does seem like a possible way forward also.
I'd be interesting to see what/how callbacks would work vs. futures and what extension point we could provide to mistral for task execution (maybe there task executor would complete by doing a call to some service, not amqp for example?). Maybe some example/POC code would help all :-) -Josh -----Original Message----- From: Ivan Melnikov <imelni...@griddynamics.com> Date: Thursday, April 3, 2014 at 12:04 PM To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org>, Joshua Harlow <harlo...@yahoo-inc.com> Subject: Re: [openstack-dev] [Mistral] How Mistral handling long running delegate tasks > >I'm trying to catch up this rather long and interesting discussion, >sorry for somewhat late reply. > >I can see aspects of 'lazy model' support in TaskFlow: >- how tasks are executed and reverted >- how flows are run >- how engine works internally > >Let me address those aspects separately. > >== Executing and reverting tasks == > >I think that should be done via different interface then running a flow >(or scheduling it to run), as it is completely different thing. In >current TaskFlow this interface is called task executor: >https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_ >engine/executor.py#L57 > >That is actually how our WorkerBasedEngine was implemented: it's the >same engine with special task executor that schedules tasks on worker >instead of running task code locally. > >Task executors are not aware of flows by design, all they do is >executing and reverting tasks. That means that task executors can be >easily shared between engines if that's wanted. > >Current TaskExecutorBase interface uses futures (PEP 3148-like). When I >proposed it, futures looked like good tool for the task at hand (see >e.g. async task etherpad >https://etherpad.openstack.org/p/async-taskflow-tasks) > >Now it may be time to reconsider that: having one future object per >running task may become a scalability issue. It may be worth to use >callbacks instead. It should not be too hard to refactor current engine >for that. Also, as TaskExecutorBase is an internal API, there should not >be any compatibility issues. > >Then, we can make task executor interface public and allow clients to >provide their own task executors. It will be possible then for Mistral >to implement its own task executor, or several, and share the >executors between all the engine instances. > >You can call it a plan;) > >== Running flows == > >To run the flow TaskFlow client uses engine interface; also, there are >few of helper functions provided for convenience: > >http://docs.openstack.org/developer/taskflow/engines.html#module-taskflow. >engines.base >http://docs.openstack.org/developer/taskflow/engines.html#creating-engines > >That is part of our public API, it is stable and good enough. Basically, >I don't think this API needs any major change. > >Maybe it worth to add function or method to schedule running flow >without actually waiting for flow completion (at least, it was on my top >secret TODO list for quite a long time). > >== Engine internals == > >Each engine eats resources, like thread it runs on; using these >resources to run one flow only is somewhat wasteful. Some work is >already planned to address this situation (see e.g. >https://blueprints.launchpad.net/taskflow/+spec/share-engine-thread). >Also, it might be good idea to implement different 'type' of engine to >support 'lazy' model, as Joshua suggests. > >But whatever should and will be done about it, I daresay all that work >can be done without affecting API more then I described above. > >-- >WBR, >Ivan A. Melnikov > >... tasks must flow ... > > >On 02.04.2014 01:51, Dmitri Zimine wrote: >> Even more responses inline :) >[...] _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
