[Twisted-Python] waiting on any deferred in a list?
Hi all, I've been using a pattern for quite some time to collect a bunch o' deferreds in a list and then yield on them one at a time. Here is some untested pseudocode: --- @defer.inlineCallbacks def foo( ls ): defs = [] # get all the deferreds going for item in ls: d = some_deferred_returning_function( item ) defs.append( ( item, d ) ) # associate item with deferred # process results not as they come in but in the original order for ( item, d ) in defs: result = yield d print 'The result on item', item, 'is', result --- In this way, I can get good concurrency (by spinning up all deferreds at once) but also I can iterate on the results using a straightforward traversal of the list data structure and yield. This is working great for me. But on occasion I'd prefer that I can access the results not in the order that I originally got, but rather in the order of completion of the deferreds, so the faster ones come in first, and the slower ones don't hold up faster ones. The usual case when I'd want this is when I have "slower" deferreds near the front, and "faster" deferreds near the back. I can't help but think that there's some async data structure out there for me that can solve this. I've used DeferredList before, and I can't see that it does what I want: it gives you a choice between getting all results only when they are all done (default behavior), or getting the first one that's ready (fireOnOneCallback=True). I want all results, but I want the processing of the results to unfold as the results come in. Thanks for any ideas. -- Benjamin Rutt ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] waiting on any deferred in a list?
I think you'll want an idiom kind of like this (also untested ;):
@inlineCallbacks
def foo(ls):
defs = []
def process_an_item(result, item):
print("Item '{}' done: {}".format(item, result))
for item in ls:
d = something_async(item)
d.addCallback(process_an_item, item)
defs.append(d)
yield DeferredList(defs)
Error-handling left as an exercise for the reader :)
--
meejah
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] IPv6 comparison handling
Hello, In my UDP Twisted based library there is a problem with IPv6 addresses representation. Addresses are compared as simple strings, and it sometimes causes mismatches: "fe80:::bbff:fecc:" should be equal to "fe80:::::bbff:fecc:" I've noticed that Python 3 has nice new module "ipaddress" exactly for this kind of problems, and it has been backported to Python 2.7. Before I add it as a new dependency I wanted to ask if maybe Twisted has some other suggested solution for this problem? Best Regards Maciej Wasilak ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] IPv6 comparison handling
> On Apr 24, 2016, at 12:21 PM, Maciej Wasilak wrote:
>
> Hello,
>
> In my UDP Twisted based library there is a problem with IPv6 addresses
> representation. Addresses are compared as simple strings, and it
> sometimes causes mismatches:
>
> "fe80:::bbff:fecc:" should be equal to
> "fe80:::::bbff:fecc:"
>
> I've noticed that Python 3 has nice new module "ipaddress" exactly for
> this kind of problems, and it has been backported to Python 2.7.
> Before I add it as a new dependency I wanted to ask if maybe Twisted
> has some other suggested solution for this problem?
You should probably use ipaddress. Twisted's facilities in this area are
somewhat primitive, and should be improved; particularly, you _should_ be able
to do:
a = IPv6Address("UDP", "fe80::", 0)
b = IPv6Address("UDP", "fe80::00:00", 0)
a == b
but right now that doesn't work.
A patch that fixes it would be much appreciated though!
-glyph
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
