Hi Joshua,
You asked a lot of questions. I will try to answer.
Le lundi 7 juillet 2014, 17:41:34 Joshua Harlow a écrit :
> * Why focus on a replacement low level execution model integration instead
> of higher level workflow library or service (taskflow, mistral... other)
> integration?
I don't know tasklow, I cannot answer to this question.
How do you write such code using taskflow?
@asyncio.coroutine
def foo(self):
result = yield from some_async_op(...)
return do_stuff(result)
> * Was the heat (asyncio-like) execution model[1] examined and learned from
> before considering moving to asyncio?
I looked at Heat coroutines, but it has a design very different from asyncio.
In short, asyncio uses an event loop running somewhere in the background,
whereas Heat explicitly schedules the execution of some tasks (with
"TaskRunner"), blocks until it gets the result and then stop completly its
"event loop". It's possible to implement that with asyncio, there is for
example a run_until_complete() method stopping the event loop when a future is
done. But asyncio event loop is designed to run "forever", so various projects
can run tasks "at the same time", not only a very specific section of the code
to run a set of tasks.
asyncio is not only designed to schedule callbacks, it's also designed to
manager file descriptors (especially sockets). It can also spawn and manager
subprocessses. This is not supported by Heat scheduler.
IMO Heat scheduler is too specific, it cannot be used widely in OpenStack.
> * A side-question, how do asyncio and/or trollius support debugging, do
> they support tracing individual co-routines? What about introspecting the
> state a coroutine has associated with it? Eventlet at least has
> http://eventlet.net/doc/modules/debug.html (which is better than nothing);
> does an equivalent exist?
asyncio documentation has a section dedicated to debug:
https://docs.python.org/dev/library/asyncio-dev.html
asyncio.Task has get_stack() and print_stack() methods can be used to get the
current state of a task. I don't know if it is what you are looking for.
I modified recently asyncio Task and Handle to save the traceback where they
were created, it's now much easier to debug code. asyncio now also logs "slow
callbacks" which may block the event loop (reduce the reactivity).
Read: We are making progress to ease the development and debug asyncio code.
I don't know exactly what you expect for debugging. However, there is no
"tracing" feature" in asyncio yet.
> This is the part that I really wonder about. Since asyncio isn't just a
> drop-in replacement for eventlet (which hid the async part under its
> *black magic*), I very much wonder how the community will respond to this
> kind of mindset change (along with its new *black magic*).
I disagree with you, asyncio is not "black magic". It's well defined. The
execution of a coroutine is complex, but it doesn't use magic. IMO eventlet
task switching is more black magic, it's not possible to guess it just by
reading the code.
Sorry but asyncio is nothing new :-( It's just a fresh design based on
previous projects.
Python has Twisted since more than 10 years. More recently, Tornado came. Both
support coroutines using "yield" (see @inlineCallbacks and toro). Thanks to
these two projects, there are already libraries which have an async API, using
coroutines or callbacks.
These are just the two major projects, they are much more projects, but they
are smaller and younger.
Other programming languages are also moving to asynchronous programming. Read
for example this article which lists some of them:
https://glyph.twistedmatrix.com/2014/02/unyielding.html
> * Is the larger python community ready for this?
>
> Seeing other responses for supporting libraries that aren't asyncio
> compatible it doesn't inspire confidence that this path is ready to be
> headed down. Partially this is due to the fact that its a completely new
> programming model and alot of underlying libraries will be forced to
> change to accommodate this (sqlalchemy, others listed in [5]...).
The design of the asyncio core is really simple, it's mostly based on
callbacks. Callbacks is an old concept, and many libraries already accept a
callback to send the result when they are done.
In asyncio, you can use loop.call_soon(callback) to schedule the execution of
a callback, and future.add_done_callback() to connect a future to a callback.
Oh by way, the core of Twisted and Tornado also uses callbacks. Using
callbacks makes you compatible with Twisted, Tornado and asyncio.
asyncio already has projects compatible with it, see this list:
https://code.google.com/p/tulip/wiki/ThirdParty
There are for example Redis, PostgreSQL and MongoDB drivers. There is also an
AMQP implementation.
You are not forced to make your code async. You can run blocking code in
threads (a pool of threads) using loop.run_in_executor(). For example, DNS
resolution is currently done like that (the socket.getaddrinfo() function is
blocking). Or you also just block until you get the result, in some cases it
does not impact performances (ex: CLI client waiting for the answer of a
server).
> Along a related question,
> seeing that openstack needs to support py2.x and py3.x will this mean that
> trollius will be required to be used in 3.x (as it is the least common
> denominator, not new syntax like 'yield from' that won't exist in 2.x).
> Does this mean that libraries that will now be required to change will be
> required to use trollius (the pulsar[6] framework seemed to mesh these two
> nicely); is this understood by those authors?
It *is* possible to write code working on asyncio and trollius:
http://trollius.readthedocs.org/#write-code-working-on-trollius-and-tulip
They are different options for that. They are already projects supporting
asyncio and trollius.
> Is this the direction we
> want to go down (if we stay focused on ensuring py3.x compatible, then why
> not just jump to py3.x in the first place)?
FYI OpenStack does not support Python 3 right now. I'm working on porting
OpenStack to Python 3, we made huge progress, but it's not done yet.
Anyway, the new RHEL 7 release doesn't provide Python 3.3 in the default
system, you have to enable the SCL repository (which provides Python 3.3). And
Python 2.7 or even 2.6 is still used in production.
I would also prefer to use directly "yield from" and "just" drop Python 2
support. But dropping Python 2 support is not going to happen before at least
2 years.
We should port OpenStack to Python 3 right now!
http://techs.enovance.com/6521/openstack_python3
Status of the port to Python 3:
https://wiki.openstack.org/wiki/Python3
I hope that I replied to some of your questions.
Victor
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
