[Twisted-Python] deferred generators in inline callbacks

[Алексей Кирпичников]

Hi.

I have a question about inline callbacks usage.

Let's assume I have an iterator that works synchronously. I want to
wrap this generator to work asynchronously in threads:

def some_generator():
cursor = make_iterator() # cursor's 'next' method uses some nasty
blocking I/O
while 1:
yield deferToThread(cursor.next)


Next, I want to asynchronously iterate over this generator in
inlineCallbacks semantics. I should say:

@inlineCallbacks
def do_stuff():
for item in some_generator():
try:
real_item = yield item
# do stuff with real_item
except StopIteration:
break


It somewhat annoys me that I should always convert item to real_item
so inlineCallbacks can properly asynchronize my code. I also need to
implement StopIteration myself :(
Is there any easier way to do this in Twisted?

Thanks in advance!

-- 
Alex

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] deferred generators in inline callbacks

[exarkun]

On 06:24 pm, alex.k...@gmail.com wrote:
>Hi.
>
>I have a question about inline callbacks usage.
>
>Let's assume I have an iterator that works synchronously. I want to
>wrap this generator to work asynchronously in threads:
>
>def some_generator():
>cursor = make_iterator() # cursor's 'next' method uses some nasty
>blocking I/O
>while 1:
>yield deferToThread(cursor.next)
>
>
>Next, I want to asynchronously iterate over this generator in
>inlineCallbacks semantics. I should say:
>
>@inlineCallbacks
>def do_stuff():
>for item in some_generator():
>try:
>real_item = yield item
># do stuff with real_item
>except StopIteration:
>break
>
>
>It somewhat annoys me that I should always convert item to real_item
>so inlineCallbacks can properly asynchronize my code. I also need to
>implement StopIteration myself :(
>Is there any easier way to do this in Twisted?

You're basically looking for coroutines - ie, context switching across 
multiple stack frames (from inside some_generator through do_stuff out 
to the implementation of inlineCallbacks).

There are several coroutine libraries for Python.  Greenlets is probably 
the most popular.  You can use it with Twisted, if you want.

Personally, I'd probably go for something more like:

def worker(item):
# do stuff with item

hook_up(some_generator(), worker)

with some kind of hook_up that does the parts of this task that you're 
tired of repeating over and over again (like turning item into 
real_item).

The result doesn't require figuring out how your coroutines are 
affecting control flow and probably gives you something more composable 
too.

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [Twisted] #5437: sys.settrace(None) in process.py raises RuntimeError in debugger

[Bob Novas]

Well, what wing's debugger does is to raise RuntimeError, and NOT CONTINUE
to execute that path. So the child never runs. That's bad!  What more can I
say?  I have to comment out that line every time I run on a new installation
if I want to debug in linux (Windows doesn't go down that code path).  Bob

> -Original Message-
> From: Twisted [mailto:t...@twistedmatrix.com]
> Sent: Tuesday, December 20, 2011 4:40 PM
> Subject: Re: [Twisted] #5437: sys.settrace(None) in process.py raises
> RuntimeError in debugger
> 
> #5437: sys.settrace(None) in process.py raises RuntimeError in debugger
> --+-
> --+
>  Reporter:  BobNovas  |   Owner:  BobNovas
>  Type:  defect|  Status:  new
>  Priority:  normal|   Milestone:
> Component:  core  |Keywords:  settrace, process
>Branch:|   Branch_author:
> Launchpad_bug:|
> --+-
> --+
> Changes (by exarkun):
> 
>   * owner:  => BobNovas
>   * milestone:  regular-releases =>
>   * branch_author:  BobNovas =>
> 
> 
> Comment:
> 
>  The ''regular-releases'' milestone is for release automation.  The
''author'' field
> is for svn branch authorship.  I've unset these two  fields.
> 
>  I'm think the intent of the `sys.settrace(None)` call is to disable
debuggers.  It
> does this because there's very little left to do before the  Python
process is going
> to be replaced by a new process (which will not be  debuggable using the
trace
> hook) and because there are two different  Python processes being debugged
at
> that point in the code, because of the  `os.fork` call which has succeeded
just
> beforehand.
> 
>  Does Wing have some way to meaningfully debug two processes
> simultaneously, where the second is created using `os.fork` in the first?
>  If not, then there's no use in removing the `sys.settrace(None)`, because
it will
> just break in some other way.
> 
>  I have very little experience with Wing's debugger, so I don't know what
it's
> possible to do in this case.  I'm also not really sure what a Wing
debugger user
> (ie you :) expect to happen.
> 
>  Can you shed some more light on this?  Thanks.
> 
> --
> Ticket URL: 
> Twisted  Engine of your Internet


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python