So I've been thinking how to respond to this email, and here goes (shields up!),
First things first; thanks mark and victor for the detailed plan and making it visible to all. It's very nicely put together and the amount of thought put into it is great to see. I always welcome an effort to move toward a new structured & explicit programming model (which asyncio clearly helps make possible and strongly encourages/requires). So now to some questions that I've been thinking about how to address/raise/ask (if any of these appear as FUD, they were not meant to be): * Why focus on a replacement low level execution model integration instead of higher level workflow library or service (taskflow, mistral... other) integration? Since pretty much all of openstack is focused around workflows that get triggered by some API activated by some user/entity having a new execution model (asyncio) IMHO doesn't seem to be shifting the needle in the direction that improves the scalability, robustness and crash-tolerance of those workflows (and the associated projects those workflows are currently defined & reside in). I *mostly* understand why we want to move to asyncio (py3, getting rid of eventlet, better performance? new awesomeness...) but it doesn't feel that important to actually accomplish seeing the big holes that openstack has right now with scalability, robustness... Let's imagine a different view on this; if all openstack projects declaratively define the workflows there APIs trigger (nova is working on task APIs, cinder is getting there to...), and in the future when the projects are *only* responsible for composing those workflows and handling the API inputs & responses then the need for asyncio or other technology can move out from the individual projects and into something else (possibly something that is being built & used as we speak). With this kind of approach the execution model can be an internal implementation detail of the workflow 'engine/processor' (it will also be responsible for fault-tolerant, robust and scalable execution). If this seems reasonable, then why not focus on integrating said thing into openstack and move the projects to a model that is independent of eventlet, asyncio (or the next greatest thing) instead? This seems to push the needle in the right direction and IMHO (and hopefully others opinions) has a much bigger potential to improve the various projects than just switching to a new underlying execution model. * Was the heat (asyncio-like) execution model[1] examined and learned from before considering moving to asyncio? I will try not to put words into the heat developers mouths (I can't do it justice anyway, hopefully they can chime in here) but I believe that heat has a system that is very similar to asyncio and coroutines right now and they are actively moving to a different model due to problems in part due to using that coroutine model in heat. So if they are moving somewhat away from that model (to a more declaratively workflow model that can be interrupted and converged upon [2]) why would it be beneficial for other projects to move toward the model they are moving away from (instead of repeating the issues the heat team had with coroutines, ex, visibility into stack/coroutine state, scale limitations, interruptibility...)? * A side-question, how do asyncio and/or trollius support debugging, do they support tracing individual co-routines? What about introspecting the state a coroutine has associated with it? Eventlet at least has http://eventlet.net/doc/modules/debug.html (which is better than nothing); does an equivalent exist? * What's the current thinking on avoiding the chaos (code-change-wise and brain-power-wise) that will come from a change to asyncio? This is the part that I really wonder about. Since asyncio isn't just a drop-in replacement for eventlet (which hid the async part under its *black magic*), I very much wonder how the community will respond to this kind of mindset change (along with its new *black magic*). Will the TC/foundation offer training, tutorials... on the change that this brings? Should the community even care? If we say just focus on workflows & let the workflow 'engine/processor' do the dirty work; then I believe the community really doesn't need to care (and rightfully so) about how their workflows get executed (either by taskflow, mistral, pigeons...). I believe this seems like a fair assumption to make; it could even be reinforced (I am not an expert here) with the defcore[4] work that seems to be standardizing the integration tests that verify those workflows (and associated APIs) act as expected in the various commercial implementations. * Is the larger python community ready for this? Seeing other responses for supporting libraries that aren't asyncio compatible it doesn't inspire confidence that this path is ready to be headed down. Partially this is due to the fact that its a completely new programming model and alot of underlying libraries will be forced to change to accommodate this (sqlalchemy, others listed in [5]...). Do others feel it's appropriate to start this at this time, or does it feel premature? Of course we have to start somewhere but I start to wonder if effort is better spent elsewhere (see above). Along a related question, seeing that openstack needs to support py2.x and py3.x will this mean that trollius will be required to be used in 3.x (as it is the least common denominator, not new syntax like 'yield from' that won't exist in 2.x). Does this mean that libraries that will now be required to change will be required to use trollius (the pulsar[6] framework seemed to mesh these two nicely); is this understood by those authors? Is this the direction we want to go down (if we stay focused on ensuring py3.x compatible, then why not just jump to py3.x in the first place)? Anyways just some things to think about & discuss (from an obviously workflow-biased[7] viewpoint), Thoughts? -Josh [1] https://github.com/openstack/heat/blob/master/heat/engine/scheduler.py [2] https://review.openstack.org/#/c/95907/ [3] https://etherpad.openstack.org/p/heat-workflow-vs-convergence [4] https://wiki.openstack.org/wiki/Governance/CoreDefinition [5] https://github.com/openstack/requirements/blob/master/global-requirements.t xt [6] http://pythonhosted.org/pulsar/ [7] http://docs.openstack.org/developer/taskflow/ -----Original Message----- From: Mark McLoughlin <mar...@redhat.com> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org> Date: Thursday, July 3, 2014 at 8:27 AM To: "openstack-dev@lists.openstack.org" <openstack-dev@lists.openstack.org> Subject: [openstack-dev] [oslo] Asyncio and oslo.messaging >Hey > >This is an attempt to summarize a really useful discussion that Victor, >Flavio and I have been having today. At the bottom are some background >links - basically what I have open in my browser right now thinking >through all of this. > >We're attempting to take baby-steps towards moving completely from >eventlet to asyncio/trollius. The thinking is for Ceilometer to be the >first victim. > >Ceilometer's code is run in response to various I/O events like REST API >requests, RPC calls, notifications received, etc. We eventually want the >asyncio event loop to be what schedules Ceilometer's code in response to >these events. Right now, it is eventlet doing that. > >Now, because we're using eventlet, the code that is run in response to >these events looks like synchronous code that makes a bunch of >synchronous calls. For example, the code might do some_sync_op() and >that will cause a context switch to a different greenthread (within the >same native thread) where we might handle another I/O event (like a REST >API request) while we're waiting for some_sync_op() to return: > > def foo(self): > result = some_sync_op() # this may yield to another greenlet > return do_stuff(result) > >Eventlet's infamous monkey patching is what make this magic happen. > >When we switch to asyncio's event loop, all of this code needs to be >ported to asyncio's explicitly asynchronous approach. We might do: > > @asyncio.coroutine > def foo(self): > result = yield from some_async_op(...) > return do_stuff(result) > >or: > > @asyncio.coroutine > def foo(self): > fut = Future() > some_async_op(callback=fut.set_result) > ... > result = yield from fut > return do_stuff(result) > >Porting from eventlet's implicit async approach to asyncio's explicit >async API will be seriously time consuming and we need to be able to do >it piece-by-piece. > >The question then becomes what do we need to do in order to port a >single oslo.messaging RPC endpoint method in Ceilometer to asyncio's >explicit async approach? > >The plan is: > > - we stick with eventlet; everything gets monkey patched as normal > > - we register the greenio event loop with asyncio - this means that > e.g. when you schedule an asyncio coroutine, greenio runs it in a > greenlet using eventlet's event loop > > - oslo.messaging will need a new variant of eventlet executor which > knows how to dispatch an asyncio coroutine. For example: > > while True: > incoming = self.listener.poll() > method = dispatcher.get_endpoint_method(incoming) > if asyncio.iscoroutinefunc(method): > result = method() > self._greenpool.spawn_n(incoming.reply, result) > else: > self._greenpool.spawn_n(method) > > it's important that even with a coroutine endpoint method, we send > the reply in a greenthread so that the dispatch greenthread doesn't > get blocked if the incoming.reply() call causes a greenlet context > switch > > - when all of ceilometer has been ported over to asyncio coroutines, > we can stop monkey patching, stop using greenio and switch to the > asyncio event loop > > - when we make this change, we'll want a completely native asyncio > oslo.messaging executor. Unless the oslo.messaging drivers support > asyncio themselves, that executor will probably need a separate > native thread to poll for messages and send replies. > >If you're confused, that's normal. We had to take several breaks to get >even this far because our brains kept getting fried. > >HTH, >Mark. > >Victor's excellent docs on asyncio and trollius: > > https://docs.python.org/3/library/asyncio.html > http://trollius.readthedocs.org/ > >Victor's proposed asyncio executor: > > https://review.openstack.org/70948 > >The case for adopting asyncio in OpenStack: > > https://wiki.openstack.org/wiki/Oslo/blueprints/asyncio > >A previous email I wrote about an asyncio executor: > > http://lists.openstack.org/pipermail/openstack-dev/2013-June/009934.html > >The mock-up of an asyncio executor I wrote: > > >https://github.com/markmc/oslo-incubator/blob/8509b8b/openstack/common/mes >saging/_executors/impl_tulip.py > >My blog post on async I/O and Python: > > http://blogs.gnome.org/markmc/2013/06/04/async-io-and-python/ > >greenio - greelets support for asyncio: > > https://github.com/1st1/greenio/ > > >_______________________________________________ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
