[Twisted-Python] Returning a deferred from buildProtocol t.i.p.Factory
Hi all
I'm building a simple TCP load balancer based on a code snippet from Glyph
on SO:
http://stackoverflow.com/questions/4096061/general-question-regarding-wether-or-not-use-twisted-in-tcp-proxy-project
It's served me well but I can't work out how to convert Glyphs round robin
retrieval of the server endpoint into an async balancing decision in the
buildProtocol method of the Factory. If I return a deferred here it fails
with an AttributeError: Deferred instance has no attribute 'makeConnection'.
Currently I'm working around this by running a separate management loop
that periodically updates a dictionary with all the data necessary to make
my routing decision so that I can do it without a deferred. This worries me
because I may be making my decision on slightly stale data and I'd really
like this to be a real time decision as the connection comes in. Does
anyone have a clever way of doing this?
An example is below. The hashed out buildProtocol is a synchronous decision
which works. Thanks in advance!
from twisted.internet.protocol import Factory
from twisted.protocols.portforward import ProxyFactory
from twisted.internet import reactor, defer
import random
from twisted.python import log
import sys
log.startLogging(sys.stderr)
local_ports = set([1024, 1025])
def port_routing_decision_sync():
return random.choice(list(local_ports))
def port_routing_decision_async():
d = defer.Deferred()
reactor.callLater(1, d.callback, port_routing_decision_sync())
return d
class Balancer(Factory):
# def buildProtocol(self, addr):
# port = port_routing_decision_sync()
# print "connecting to local port {}".format(port)
# return ProxyFactory("127.0.0.1", port).buildProtocol(addr)
@defer.inlineCallbacks
def buildProtocol(self, addr):
port = yield port_routing_decision_async()
print "connecting to local port {}".format(port)
defer.returnValue(ProxyFactory("127.0.0.1",
port).buildProtocol(addr))
def main():
factory = Balancer()
reactor.listenTCP(5678, factory)
reactor.run()
if __name__ == "__main__":
main()
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Returning a deferred from buildProtocol t.i.p.Factory
Hi Tom, On 11/16/13 8:09 AM, Tom van Neerijnen wrote: Hi all I'm building a simple TCP load balancer based on a code snippet from Glyph on SO: http://stackoverflow.com/questions/4096061/general-question-regarding-wether-or-not-use-twisted-in-tcp-proxy-project It's served me well but I can't work out how to convert Glyphs round robin retrieval of the server endpoint into an async balancing decision in the buildProtocol method of the Factory. If I return a deferred here it fails with an AttributeError: Deferred instance has no attribute 'makeConnection'. [SNIP] Have you considered using https://pypi.python.org/pypi/txLoadBalancer as the basis for your load-balancer? It supports random, round-robin, least-connections, and weighted, so perhaps it would suit your needs. Hope this helps, L. Daniel Burr ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Ticket #1330 - Socks V5 functionality
Hi, I'd like to help out and write unit tests for the Socks v5 code in this ticket: https://twistedmatrix.com/trac/ticket/1330 Should I write something very similar to this?? :: http://twistedmatrix.com/trac/browser/trunk/twisted/test/test_socks.py My goal is getting socksv5 client and server code merged to mainline Twisted with unit tests. Cheers, David ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Returning a deferred from buildProtocol t.i.p.Factory
On Nov 16, 2013, at 7:09 AM, Tom van Neerijnen wrote:
> Hi all
>
> I'm building a simple TCP load balancer based on a code snippet from Glyph on
> SO:
> http://stackoverflow.com/questions/4096061/general-question-regarding-wether-or-not-use-twisted-in-tcp-proxy-project
>
> It's served me well but I can't work out how to convert Glyphs round robin
> retrieval of the server endpoint into an async balancing decision in the
> buildProtocol method of the Factory. If I return a deferred here it fails
> with an AttributeError: Deferred instance has no attribute 'makeConnection'.
>
> Currently I'm working around this by running a separate management loop that
> periodically updates a dictionary with all the data necessary to make my
> routing decision so that I can do it without a deferred. This worries me
> because I may be making my decision on slightly stale data and I'd really
> like this to be a real time decision as the connection comes in. Does anyone
> have a clever way of doing this?
>
Hi Tom,
One possibly unexpected aspect of using @inlineCallbacks is that the decorated
function itself returns a Deferred. This is why you see the
AttributeError...the machinery calling buildProtocol expects an IProtocol
instance (or None), and the function is returning a Deferred.
`defer.returnValue()` is provided to the callback on that Deferred, not as a
direct return value from the decorated function.
If you want to make the routing decision when the client connects, then you
could push the decision-making process down into the Protocol itself.
Here's a quick mockup overriding connectionMade in a ProxyServer protocol
subclass. It calls the factory routing function (which may or may not return a
deferred), and connects the proxy once the decision has been made.
from twisted.internet.protocol import Factory
from twisted.protocols.portforward import ProxyServer
class Balancer(Factory):
protocol = RoutingProxyServer
routing_func = port_routing_decision_async
class RoutingProxyServer(ProxyServer):
def connectionMade(self):
# Don't read anything from the connecting client until we have
# somewhere to send it to.
self.transport.pauseProducing()
client = self.clientProtocolFactory()
client.setServer(self)
if self.reactor is None:
from twisted.internet import reactor
self.reactor = reactor
def connectProxy(host, port):
self.reactor.connectTCP(host, port, client)
d = maybeDeferred(self.factory.routing_func)
d.addCallback(connectProxy)
d.addErrback(log.err)
Lucas
> An example is below. The hashed out buildProtocol is a synchronous decision
> which works. Thanks in advance!
>
> from twisted.internet.protocol import Factory
> from twisted.protocols.portforward import ProxyFactory
> from twisted.internet import reactor, defer
> import random
>
> from twisted.python import log
> import sys
> log.startLogging(sys.stderr)
>
> local_ports = set([1024, 1025])
>
> def port_routing_decision_sync():
> return random.choice(list(local_ports))
>
> def port_routing_decision_async():
> d = defer.Deferred()
> reactor.callLater(1, d.callback, port_routing_decision_sync())
> return d
>
> class Balancer(Factory):
> # def buildProtocol(self, addr):
> # port = port_routing_decision_sync()
> # print "connecting to local port {}".format(port)
> # return ProxyFactory("127.0.0.1", port).buildProtocol(addr)
>
> @defer.inlineCallbacks
> def buildProtocol(self, addr):
> port = yield port_routing_decision_async()
> print "connecting to local port {}".format(port)
> defer.returnValue(ProxyFactory("127.0.0.1", port).buildProtocol(addr))
>
> def main():
> factory = Balancer()
> reactor.listenTCP(5678, factory)
> reactor.run()
>
> if __name__ == "__main__":
> main()
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Ticket #1330 - Socks V5 functionality
On 06:14 pm, dstainton...@gmail.com wrote: Hi, I'd like to help out and write unit tests for the Socks v5 code in this ticket: https://twistedmatrix.com/trac/ticket/1330 Should I write something very similar to this?? :: http://twistedmatrix.com/trac/browser/trunk/twisted/test/test_socks.py My goal is getting socksv5 client and server code merged to mainline Twisted with unit tests. twisted/test/test_socks.py is a bad example of a test suite. Here are the things about it you should not emulate: * It has documentation that is far from complete. Documentation is just as important in unit tests as elsewhere. In particular, documenting the intent of every test method is critical otherwise the test suite is very difficult to maintain. * It exercises too much code in each test method. Well written test methods do a single very simple thing. A good rule of thumb is that there should only be one `TestCase.assert...` method call in each test method. * It uses some `TestCase.assert...` methods which are deprecated or soon to be deprecated. `assert_` is the main offender here. * It doesn't completely cover the implementation (probably because the implementation wasn't developed test-driven). You can achieve full test coverage without doing test-driven development but it takes more discipline. I suggest doing a test-driven implementation of the SOCKSv5 functionality you want (the easy approach to this is to start writing tests, then copy the *smallest* possible piece of the existing, untested implementation into your new implementation to make those tests pass; repeat until you have all of the desired functionality). * `StringTCPTransport` seems redundant. `StringTransport` offers all of this functionality already. * Many names used in the module don't follow the Twisted name convention (most obviously, "under_scores" are used throughout rather than "camelCase"). * Native strings are used to represent byte strings throughout. * The protocol interface is uniformly misused (it should call `makeConnection` not `connectionMade`) Hope this helps, Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Ticket #1330 - Socks V5 functionality
Hi Jean-Paul, Thanks for all the info, observations about the existing code and for the coding advice! I think I should only implement the socks 5 server side since txsocksx seems to have the client implementation covered. Some of the Tor developers use it... I'm not used to test driven development. I'll give it a try and implement the SOCKSv5 server functionality... Cheers! David On Sat, Nov 16, 2013 at 11:44 AM, wrote: > On 06:14 pm, dstainton...@gmail.com wrote: >> >> Hi, I'd like to help out and write unit tests for the Socks v5 code in >> this ticket: >> https://twistedmatrix.com/trac/ticket/1330 >> >> Should I write something very similar to this?? :: >> http://twistedmatrix.com/trac/browser/trunk/twisted/test/test_socks.py >> >> My goal is getting socksv5 client and server code merged to mainline >> Twisted with unit tests. > > > > twisted/test/test_socks.py is a bad example of a test suite. Here are the > things about it you should not emulate: > > * It has documentation that is far from complete. Documentation is just as > important in unit tests as elsewhere. In particular, documenting the intent > of every test method is critical otherwise the test suite is very difficult > to maintain. > > * It exercises too much code in each test method. Well written test > methods do a single very simple thing. A good rule of thumb is that there > should only be one `TestCase.assert...` method call in each test method. > > * It uses some `TestCase.assert...` methods which are deprecated or soon to > be deprecated. `assert_` is the main offender here. > > * It doesn't completely cover the implementation (probably because the > implementation wasn't developed test-driven). You can achieve full test > coverage without doing test-driven development but it takes more discipline. > I suggest doing a test-driven implementation of the SOCKSv5 functionality > you want (the easy approach to this is to start writing tests, then copy the > *smallest* possible piece of the existing, untested implementation into your > new implementation to make those tests pass; repeat until you have all of > the desired functionality). > > * `StringTCPTransport` seems redundant. `StringTransport` offers all of > this functionality already. > > * Many names used in the module don't follow the Twisted name convention > (most obviously, "under_scores" are used throughout rather than > "camelCase"). > > * Native strings are used to represent byte strings throughout. > > * The protocol interface is uniformly misused (it should call > `makeConnection` not `connectionMade`) > > Hope this helps, > Jean-Paul > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Weekly Bug Summary
Bug summary __ Summary for 2013-11-10 through 2013-11-17 Opened Closed Total Change Enhancements: 13 3 1138+10 Defects: 4 1703 +3 Tasks: 1 0 92 +1 Regressions: 0 0 1 +0 Total:18 4 1934+14 |== Type Changes |== Priority Changes |== Component Changes |Defect: +3 |Normal: +12 |Core:+5 |Enhancement: +10 |Low: +2 |Names: +7 |Task: +1 |Runner: +1 |Trial: +2 |Web: -1 Total Tickets Open Tickets New / Reopened Bugs __ = Normal = [#6821] Fix typo in TCPServerTests test case name (opened by atorkhov) (CLOSED, fixed) enhancement core http://twistedmatrix.com/trac/ticket/6821 [#6822] Deprecate the default resolverFactory in twisted.names.root.Resolver (opened by rwall) tasknames http://twistedmatrix.com/trac/ticket/6822 [#6823] Create an simpler root.Resolver bootstrap mechanism (opened by rwall) enhancement names http://twistedmatrix.com/trac/ticket/6823 [#6825] twisted.python.test.test_release fails with Twisted Web not available (opened by Arfrever) defect core http://twistedmatrix.com/trac/ticket/6825 [#6826] Pass scope IDs and flow info along when receiving datagrams (opened by habnabit) enhancement core http://twistedmatrix.com/trac/ticket/6826 [#6827] ImportError with daemonize (opened by kevwil) (CLOSED, wontfix) defect core http://twistedmatrix.com/trac/ticket/6827 [#6829] Add support for renegotiating TLS connections (opened by amluto) enhancement core http://twistedmatrix.com/trac/ticket/6829 [#6831] twisted/runner/portmap.c extension module unnecessarily complicates build/install process (opened by exarkun) defect runner http://twistedmatrix.com/trac/ticket/6831 [#6833] Port twisted.protocols.amp to Python 3 (opened by habnabit) enhancement core http://twistedmatrix.com/trac/ticket/6833 [#6834] SynchronousTestCase should fail when returning a Deferred from a test method (opened by lvh) enhancement trial http://twistedmatrix.com/trac/ticket/6834 [#6835] NSEC record class and lookup methods (opened by rwall) enhancement names http://twistedmatrix.com/trac/ticket/6835 [#6836] DS record class and lookup methods (opened by rwall) enhancement names http://twistedmatrix.com/trac/ticket/6836 [#6837] NSEC3 record class and lookup methods (opened by rwall) enhancement names http://twistedmatrix.com/trac/ticket/6837 [#6838] NSEC3param record class and lookup methods (opened by rwall) enhancement names http://twistedmatrix.com/trac/ticket/6838 [#6839] t.n.dns.DNSProtocol and DNSDatagramProtocol should allow an alternative Message parser to be used (opened by rwall) enhancement names http://twistedmatrix.com/trac/ticket/6839 [#6840] t.n.client.Resolver should allow alternative DNS protocol factories to be used (opened by rwall) enhancement names http://twistedmatrix.com/trac/ticket/6840 = Low = [#6828] Use ReactorBuilder for the IPv4 UDP tests (opened by habnabit) enhancement core http://twistedmatrix.com/trac/ticket/6828 [#6832] unittest 's test runner has a different criteria for test case names than trial (opened by Zancas) defect trial http://twistedmatrix.com/trac/ticket/6832 Closed Bugs __ = Normal = [#6821] Fix typo in TCPServerTests test case name (opened by atorkhov, closed by exarkun, fixed) enhancement core http://twistedmatrix.com/trac/ticket/6821 [#6827] ImportError with daemonize (opened by kevwil, closed by exarkun, wontfix) defect core http://twistedmatrix.com/trac/ticket/6827 [#4252] twisted.web.iweb.IRequest does not have a "content" attribute (opened by khorn, closed by exarkun, duplicate) enhancement webhttp://twistedmatrix.com/trac/ticket/4252 [#6095] Get rid of the circular dependency between root and client in twisted.names (opened by exarkun, closed by rwall, fixed) enhancement names http://twistedmatrix.com/trac/ticket/6095 Ticket Lifetime Stats __ Oldest open ticket - [#50] conch command-line client doesn't work in win32 (since 2003-07-12 14:41:06). Newest open ticket - [#6840] t.n.client.Resolver should allow alternative DNS protocol factories to be used (since 2013-11-14 11:00:58). Mean open ticket age: 1270 days, 7:18:56.540325. Median
