On 07/03/2014 05:27 PM, Mark McLoughlin wrote:
> Hey
>
> This is an attempt to summarize a really useful discussion that Victor,
> Flavio and I have been having today. At the bottom are some background
> links - basically what I have open in my browser right now thinking
> through all of this.
>
> We're attempting to take baby-steps towards moving completely from
> eventlet to asyncio/trollius. The thinking is for Ceilometer to be the
> first victim.
>
> </snip>
>
> When we switch to asyncio's event loop, all of this code needs to be
> ported to asyncio's explicitly asynchronous approach. We might do:
>
> @asyncio.coroutine
> def foo(self):
> result = yield from some_async_op(...)
> return do_stuff(result)
>
> or:
>
> @asyncio.coroutine
> def foo(self):
> fut = Future()
> some_async_op(callback=fut.set_result)
> ...
> result = yield from fut
> return do_stuff(result)
>
> Porting from eventlet's implicit async approach to asyncio's explicit
> async API will be seriously time consuming and we need to be able to do
> it piece-by-piece.
>
> The question then becomes what do we need to do in order to port a
> single oslo.messaging RPC endpoint method in Ceilometer to asyncio's
> explicit async approach?
>
> The plan is:
>
> - we stick with eventlet; everything gets monkey patched as normal
>
> - we register the greenio event loop with asyncio - this means that
> e.g. when you schedule an asyncio coroutine, greenio runs it in a
> greenlet using eventlet's event loop
>
> - oslo.messaging will need a new variant of eventlet executor which
> knows how to dispatch an asyncio coroutine. For example:
>
> while True:
> incoming = self.listener.poll()
> method = dispatcher.get_endpoint_method(incoming)
> if asyncio.iscoroutinefunc(method):
> result = method()
> self._greenpool.spawn_n(incoming.reply, result)
> else:
> self._greenpool.spawn_n(method)
>
> it's important that even with a coroutine endpoint method, we send
> the reply in a greenthread so that the dispatch greenthread doesn't
> get blocked if the incoming.reply() call causes a greenlet context
> switch
>
> - when all of ceilometer has been ported over to asyncio coroutines,
> we can stop monkey patching, stop using greenio and switch to the
> asyncio event loop
>
> - when we make this change, we'll want a completely native asyncio
> oslo.messaging executor. Unless the oslo.messaging drivers support
> asyncio themselves, that executor will probably need a separate
> native thread to poll for messages and send replies.
>
> If you're confused, that's normal. We had to take several breaks to get
> even this far because our brains kept getting fried.
>
Thanks Mark for putting this all together in an approachable way. This
is really interesting work, and I wish I found out about all of this
sooner :).
When I read all of this stuff and got my head around it (took some time
:) ), a glaring drawback of such an approach, and as I mentioned on the
spec proposing it [1] is that we would not really doing asyncio, we
would just be pretending we are by using a subset of it's APIs, and
having all of the really important stuff for overall design of the code
(code that needs to do IO in the callbacks for example) and ultimately -
performance, completely unavailable to us when porting.
So in Mark's example above:
@asyncio.coroutine
def foo(self):
result = yield from some_async_op(...)
return do_stuff(result)
A developer would not need to do anything that asyncio requires like
make sure that some_async_op() registers a callback with the eventloop
(using for example event_loop.add_reader/writer methods) you could just
simply make it use a 'greened' call and things would continue working
happily. I have a feeling this will in turn have a lot of people writing
code that they don't understand, and as library writers - we are not
doing an excellent job at that point.
Now porting an OpenStack project to another IO library with completely
different design is a huge job and there is unlikely a single 'right'
way to do it, so treat this as a discussion starter, that will hopefully
give us a better understanding of the problem we are trying to tackle.
So I hacked up together a small POC of a different approach. In short -
we actually use a real asyncio selector eventloop in a separate thread,
and dispatch stuff to it when we figure out that our callback is in fact
a coroutine. More will be clear form the code so:
(Warning - hacky code ahead): [2]
I will probably be updating it - but if you just clone the repo, all the
history is there. I wrote it without the oslo.messaging abstractions
like listener and dispatcher, but it is relatively easy to see which
bits of code would go in those.
Several things worth noting as you read the above. First one is that we
do not monkeypatch until we have fired of the asyncio thread (Victor
correctly noticed this would be a problem in a comment on [1]). This may
seem hacky (and it is) but if decide to go further down this road - we
would probably not be 'greening the world' but rather importing patched
non-ported modules when we need to dispatch to them. This may sound like
a big deal, and it is, but it is critical to actually running ported
code in a real asyncio evenloop. I have not yet tested this further, but
from briefly reading eventlet code - it seems like ti should work.
Another interesting problem is (as I have briefly mentioned in [1]) -
what happens when we need to synchronize between eventlet-run and
asyncio-run callbacks while we are in the process of porting. I don't
have a good answer to that yet, but it is worth noting that the proposed
approach doesn't either, and this is a thing we should have some idea
about before going in with a knife.
Now for some marketing :) - I can see several advantages of such an
approach, the obvious one being as stated, that we are in fact doing
asyncio, so we are all in. Also as you can see [2] the implementation is
far from magical - it's (surprisingly?) simple, and requires no other
additional dependencies apart from trollius itself (granted greenio is
not too complex either). I am sure that we would hit some other problems
that were not clear from this basic POC (it was done in ~3 hours on a
bus), but it seems to me that those problems will likely need to be
solved anyhow if we are to port Ceilometer (or any other project) to
asyncio, we will just hit them sooner this way.
It was a fun approach to ponder anyway - so I am looking forward to
comments and thoughts.
Thanks,
Nikola
[1] https://review.openstack.org/#/c/104792/
[2]
https://github.com/djipko/eventlet-asyncio/blob/master/asyncio_dispatch.py
> HTH,
> Mark.
>
> Victor's excellent docs on asyncio and trollius:
>
> https://docs.python.org/3/library/asyncio.html
> http://trollius.readthedocs.org/
>
> Victor's proposed asyncio executor:
>
> https://review.openstack.org/70948
>
> The case for adopting asyncio in OpenStack:
>
> https://wiki.openstack.org/wiki/Oslo/blueprints/asyncio
>
> A previous email I wrote about an asyncio executor:
>
> http://lists.openstack.org/pipermail/openstack-dev/2013-June/009934.html
>
> The mock-up of an asyncio executor I wrote:
>
>
> https://github.com/markmc/oslo-incubator/blob/8509b8b/openstack/common/messaging/_executors/impl_tulip.py
>
> My blog post on async I/O and Python:
>
> http://blogs.gnome.org/markmc/2013/06/04/async-io-and-python/
>
> greenio - greelets support for asyncio:
>
> https://github.com/1st1/greenio/
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
