-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/07/14 05:30, Mark McLoughlin wrote: > Hey > > This is an attempt to summarize a really useful discussion that Victor, > Flavio and I have been having today. At the bottom are some background > links - basically what I have open in my browser right now thinking > through all of this. > > We're attempting to take baby-steps towards moving completely from > eventlet to asyncio/trollius. The thinking is for Ceilometer to be the > first victim.
Has this been widely agreed on? It seems to me like we are mixing two issues: 1) we need to move to py3 2) some people want to move from eventlet (I am not convinced that the volume of code changes warrants the end goal - and review load) To achieve "1)" in a lower risk change, shouldn't we rather run eventlet on top of asyncio? - i.e. not require widespread code changes. So we can maintain the main loop API but move to py3. I am not sure on the feasibility, but seems to me like a more contained change. - -Angus > > Ceilometer's code is run in response to various I/O events like REST API > requests, RPC calls, notifications received, etc. We eventually want the > asyncio event loop to be what schedules Ceilometer's code in response to > these events. Right now, it is eventlet doing that. > > Now, because we're using eventlet, the code that is run in response to > these events looks like synchronous code that makes a bunch of > synchronous calls. For example, the code might do some_sync_op() and > that will cause a context switch to a different greenthread (within the > same native thread) where we might handle another I/O event (like a REST > API request) while we're waiting for some_sync_op() to return: > > def foo(self): > result = some_sync_op() # this may yield to another greenlet > return do_stuff(result) > > Eventlet's infamous monkey patching is what make this magic happen. > > When we switch to asyncio's event loop, all of this code needs to be > ported to asyncio's explicitly asynchronous approach. We might do: > > @asyncio.coroutine > def foo(self): > result = yield from some_async_op(...) > return do_stuff(result) > > or: > > @asyncio.coroutine > def foo(self): > fut = Future() > some_async_op(callback=fut.set_result) > ... > result = yield from fut > return do_stuff(result) > > Porting from eventlet's implicit async approach to asyncio's explicit > async API will be seriously time consuming and we need to be able to do > it piece-by-piece. > > The question then becomes what do we need to do in order to port a > single oslo.messaging RPC endpoint method in Ceilometer to asyncio's > explicit async approach? > > The plan is: > > - we stick with eventlet; everything gets monkey patched as normal > > - we register the greenio event loop with asyncio - this means that > e.g. when you schedule an asyncio coroutine, greenio runs it in a > greenlet using eventlet's event loop > > - oslo.messaging will need a new variant of eventlet executor which > knows how to dispatch an asyncio coroutine. For example: > > while True: > incoming = self.listener.poll() > method = dispatcher.get_endpoint_method(incoming) > if asyncio.iscoroutinefunc(method): > result = method() > self._greenpool.spawn_n(incoming.reply, result) > else: > self._greenpool.spawn_n(method) > > it's important that even with a coroutine endpoint method, we send > the reply in a greenthread so that the dispatch greenthread doesn't > get blocked if the incoming.reply() call causes a greenlet context > switch > > - when all of ceilometer has been ported over to asyncio coroutines, > we can stop monkey patching, stop using greenio and switch to the > asyncio event loop > > - when we make this change, we'll want a completely native asyncio > oslo.messaging executor. Unless the oslo.messaging drivers support > asyncio themselves, that executor will probably need a separate > native thread to poll for messages and send replies. > > If you're confused, that's normal. We had to take several breaks to get > even this far because our brains kept getting fried. > > HTH, > Mark. > > Victor's excellent docs on asyncio and trollius: > > https://docs.python.org/3/library/asyncio.html > http://trollius.readthedocs.org/ > > Victor's proposed asyncio executor: > > https://review.openstack.org/70948 > > The case for adopting asyncio in OpenStack: > > https://wiki.openstack.org/wiki/Oslo/blueprints/asyncio > > A previous email I wrote about an asyncio executor: > > http://lists.openstack.org/pipermail/openstack-dev/2013-June/009934.html > > The mock-up of an asyncio executor I wrote: > > > https://github.com/markmc/oslo-incubator/blob/8509b8b/openstack/common/messaging/_executors/impl_tulip.py > > My blog post on async I/O and Python: > > http://blogs.gnome.org/markmc/2013/06/04/async-io-and-python/ > > greenio - greelets support for asyncio: > > https://github.com/1st1/greenio/ > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTuuLsAAoJEFrDYBLxZjWotFwH/1PYbrXFYb+gYNCWBAtGJ3mi wHdHIonIjcD3Zpnvs3o5BJDbugSlfdKzHRjSy9dQG5LyuihiFNK3AFQu/ldfJEdS /QiLIaCQYI/13AwWA9+8A/TYpoyVJytVaXm3uBp2fMhy02TYWOweh02gifUZ7CBp 3hRzGQE4EmHr0YpgQRupIDpE50GzflB8fySXyfm3Cme7vcCivyMOHexsxKe3Bm2o LxfMRPvUOqZ9MaJ6XFTy8SieNA4kMdZYmrm4pu5nrALXwxZ1dAtpD4jcjLK7Jh5T P+eZHcQTe7bYSlk+LMchfHpAlTuLTySHv/LB6wvt2ljSM4AAmABKOKwAjFFL66k= =O09e -----END PGP SIGNATURE----- _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
