Thanks clint, that was the gist of what I was getting at with the (likely to long) email.
-Josh -----Original Message----- From: Clint Byrum <cl...@fewbar.com> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org> Date: Tuesday, July 8, 2014 at 11:43 AM To: openstack-dev <openstack-dev@lists.openstack.org> Subject: Re: [openstack-dev] [oslo] Asyncio and oslo.messaging >Excerpts from Victor Stinner's message of 2014-07-08 05:47:36 -0700: >> Hi Joshua, >> >> You asked a lot of questions. I will try to answer. >> >> Le lundi 7 juillet 2014, 17:41:34 Joshua Harlow a écrit : >> > * Why focus on a replacement low level execution model integration >>instead >> > of higher level workflow library or service (taskflow, mistral... >>other) >> > integration? >> >> I don't know tasklow, I cannot answer to this question. >> >> How do you write such code using taskflow? >> >> @asyncio.coroutine >> def foo(self): >> result = yield from some_async_op(...) >> return do_stuff(result) >> > >Victor, this is a low level piece of code, which highlights the problem >that taskflow's higher level structure is meant to address. In writing >OpenStack, we want to accomplish tasks based on a number of events. Users, >errors, etc. We don't explicitly want to run coroutines, we want to >attach volumes, spawn vms, and store files. > >See this: > >http://docs.openstack.org/developer/taskflow/examples.html > >The result is consumed in the next task in the flow. Meanwhile we get >a clear definition of work-flow and very clear methods for resumption, >retry, etc. So the expression is not as tightly bound as the code above, >but that is the point, because we want to break things up into tasks >which are clearly defined and then be able to resume each one >individually. > >So what I think Josh is getting at, is that we could add asyncio support >into taskflow as an abstraction for tasks that want to be non-blocking, >and then we can focus on refactoring the code around high level work-flow >expression rather than low level asyncio and coroutines. > >> > * Was the heat (asyncio-like) execution model[1] examined and learned >>from >> > before considering moving to asyncio? >> >> I looked at Heat coroutines, but it has a design very different from >>asyncio. >> >> In short, asyncio uses an event loop running somewhere in the >>background, >> whereas Heat explicitly schedules the execution of some tasks (with >> "TaskRunner"), blocks until it gets the result and then stop completly >>its >> "event loop". It's possible to implement that with asyncio, there is >>for >> example a run_until_complete() method stopping the event loop when a >>future is >> done. But asyncio event loop is designed to run "forever", so various >>projects >> can run tasks "at the same time", not only a very specific section of >>the code >> to run a set of tasks. >> >> asyncio is not only designed to schedule callbacks, it's also designed >>to >> manager file descriptors (especially sockets). It can also spawn and >>manager >> subprocessses. This is not supported by Heat scheduler. >> >> IMO Heat scheduler is too specific, it cannot be used widely in >>OpenStack. >> > >This is sort of backwards to what Josh was suggesting. Heat can't continue >with the current approach, which is coroutine based, because we need the >the execution stack to not be in RAM on a single engine. We are going >to achieve even more concurrency than we have now through an even higher >level of task abstraction as part of the move to a convergence model. We >will likely use task-flow to express these tasks so that they are more >resumable and generally resilient to failure. > >> > Along a related question, >> > seeing that openstack needs to support py2.x and py3.x will this mean >>that >> > trollius will be required to be used in 3.x (as it is the least common >> > denominator, not new syntax like 'yield from' that won't exist in >>2.x). >> > Does this mean that libraries that will now be required to change >>will be >> > required to use trollius (the pulsar[6] framework seemed to mesh >>these two >> > nicely); is this understood by those authors? >> >> It *is* possible to write code working on asyncio and trollius: >> >>http://trollius.readthedocs.org/#write-code-working-on-trollius-and-tulip >> >> They are different options for that. They are already projects >>supporting >> asyncio and trollius. >> >> > Is this the direction we >> > want to go down (if we stay focused on ensuring py3.x compatible, >>then why >> > not just jump to py3.x in the first place)? >> >> FYI OpenStack does not support Python 3 right now. I'm working on >>porting >> OpenStack to Python 3, we made huge progress, but it's not done yet. >> >> Anyway, the new RHEL 7 release doesn't provide Python 3.3 in the >>default >> system, you have to enable the SCL repository (which provides Python >>3.3). And >> Python 2.7 or even 2.6 is still used in production. >> >> I would also prefer to use directly "yield from" and "just" drop Python >>2 >> support. But dropping Python 2 support is not going to happen before at >>least >> 2 years. >> > >Long term porting is important, however, we have immediate needs for >improvements in resilience and scalability. We cannot hang _any_ of that >on Python 3. > >_______________________________________________ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
