[Twisted-Python] Traceback on loseConnection
Hi all My users are complaining that they get the odd disconnect while using my site built with nevow/athena on FreeBSD 7.1 with the poll reactor. I recently switched to twisted 8.2 release and am now observing 199 'Unhandled Error' out of 298436 logged requests with the following traceback. Any clues? TIA, Werner 2009-02-04 05:21:29+0100 [ProxyClient,client] Unhandled Error Traceback (most recent call last): File "/usr/local/lib/python2.5/site-packages/Twisted-8.2.0_r26128-py2.5-freebsd-7.1-RELEASE-amd64.egg/twisted/application/app.py", line 348, in runReactorWithLogging reactor.run() File "/usr/local/lib/python2.5/site-packages/Twisted-8.2.0_r26128-py2.5-freebsd-7.1-RELEASE-amd64.egg/twisted/internet/base.py", line 1148, in run self.mainLoop() File "/usr/local/lib/python2.5/site-packages/Twisted-8.2.0_r26128-py2.5-freebsd-7.1-RELEASE-amd64.egg/twisted/internet/base.py", line 1160, in mainLoop self.doIteration(t) File "/usr/local/lib/python2.5/site-packages/Twisted-8.2.0_r26128-py2.5-freebsd-7.1-RELEASE-amd64.egg/twisted/internet/pollreactor.py", line 165, in doPoll log.callWithLogger(selectable, _drdw, selectable, fd, event) --- --- File "/usr/local/lib/python2.5/site-packages/Twisted-8.2.0_r26128-py2.5-freebsd-7.1-RELEASE-amd64.egg/twisted/python/log.py", line 84, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/local/lib/python2.5/site-packages/Twisted-8.2.0_r26128-py2.5-freebsd-7.1-RELEASE-amd64.egg/twisted/python/log.py", line 69, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/local/lib/python2.5/site-packages/Twisted-8.2.0_r26128-py2.5-freebsd-7.1-RELEASE-amd64.egg/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python2.5/site-packages/Twisted-8.2.0_r26128-py2.5-freebsd-7.1-RELEASE-amd64.egg/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) File "/usr/local/lib/python2.5/site-packages/Twisted-8.2.0_r26128-py2.5-freebsd-7.1-RELEASE-amd64.egg/twisted/internet/pollreactor.py", line 189, in _doReadOrWrite self._disconnectSelectable(selectable, why, inRead) File "/usr/local/lib/python2.5/site-packages/Twisted-8.2.0_r26128-py2.5-freebsd-7.1-RELEASE-amd64.egg/twisted/internet/posixbase.py", line 194, in _disconnectSelectable selectable.connectionLost(failure.Failure(why)) File "/usr/local/lib/python2.5/site-packages/Twisted-8.2.0_r26128-py2.5-freebsd-7.1-RELEASE-amd64.egg/twisted/internet/tcp.py", line 676, in connectionLost Connection.connectionLost(self, reason) File "/usr/local/lib/python2.5/site-packages/Twisted-8.2.0_r26128-py2.5-freebsd-7.1-RELEASE-amd64.egg/twisted/internet/tcp.py", line 518, in connectionLost protocol.connectionLost(reason) File "/usr/local/lib/python2.5/site-packages/Twisted-8.2.0_r26128-py2.5-freebsd-7.1-RELEASE-amd64.egg/twisted/web/http.py", line 426, in connectionLost self.handleResponseEnd() File "/usr/local/lib/python2.5/site-packages/Twisted-8.2.0_r26128-py2.5-freebsd-7.1-RELEASE-amd64.egg/twisted/web/proxy.py", line 79, in handleResponseEnd self.father.channel.transport.loseConnection() exceptions.AttributeError: 'NoneType' object has no attribute 'transport' ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] redefine signal handlers
Hello everyone, I would like to run some cleanup code when my Twisted app receives a signal (SIGINT/SIGBREAK/SIGTERM). I saw that "_SignalReactorMixin" sets the handlers and that "ReactorBase" defines the default handlers : def sigInt(self, *args): """Handle a SIGINT interrupt. """ log.msg("Received SIGINT, shutting down.") self.callFromThread(self.stop) def sigBreak(self, *args): """Handle a SIGBREAK interrupt. """ log.msg("Received SIGBREAK, shutting down.") self.callFromThread(self.stop) def sigTerm(self, *args): """Handle a SIGTERM interrupt. """ log.msg("Received SIGTERM, shutting down.") self.callFromThread(self.stop) My question is how can I redefine them, other than monkey patching or inheriting the reactor and over-riding them (which I'd rather not do since some of my code uses the windows reactor when on windows since I was having problems with windows event)? Is there such a mechanism, something like setDefaultSig("SIGINT, mySigIntHandler)? Thank you, Gabriel ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] redefine signal handlers
Hi Gabriel, On Wed, Feb 4, 2009 at 10:15 AM, Gabriel Rossetti wrote: > Hello everyone, > > I would like to run some cleanup code when my Twisted app receives a signal > (SIGINT/SIGBREAK/SIGTERM). > > I saw that "_SignalReactorMixin" sets the handlers and that "ReactorBase" > defines the default handlers : ... > My question is how can I redefine them, other than monkey patching or > inheriting the reactor and over-riding them (which I'd rather not do since > some of my code uses the windows reactor when on windows since I was having > problems with windows event)? Is there such a mechanism, something like > setDefaultSig("SIGINT, mySigIntHandler)? One easy way to do it is to reactor.run(installSignalHandlers=False) and then manually install your own signal handlers in the usual way (using the python signal library). Cheers, Reza -- Reza Lotun Senior Software Engineer Peer Technologies Limited r...@getpeer.com ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] installation help
Hi, I've taken an interest in event driven programming, I came across twisted and liked the look of it, I've tried downloading the .dmg from twistedmatrix/trac unfortunately after trying multiple installations including direct installation to both the folder where python is found (under system/frameworks...) and by changing the sys.path to another directory I fail to be able to import twisted in the interactive prompt. I have a hunch I may be missing OpenSSL (occassional errors) but I don't know enough about this (after googling) to be able to resolve this. Essentially I'm interested in writing local event-driven code, whereby I write specific objects that are capable of interaction in a more natural manner than Python would permit by threading and classes, so I'm perfectly happy with any solution that allows me to only locally program event-driven code. I would really appreciate any help whatsoever on how to resolve this installation problem! thanks, Arie ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Traceback on loseConnection
On Wed, 04 Feb 2009 11:04:26 +0100, Werner Thie wrote: Hi all My users are complaining that they get the odd disconnect while using my site built with nevow/athena on FreeBSD 7.1 with the poll reactor. I recently switched to twisted 8.2 release and am now observing 199 'Unhandled Error' out of 298436 logged requests with the following traceback. Any clues? TIA, Werner [snip] File "/usr/local/lib/python2.5/site-packages/Twisted-8.2.0_r26128- py2.5-freebsd-7.1-RELEASE-amd64.egg/twisted/web/proxy.py", line 79, in handleResponseEnd self.father.channel.transport.loseConnection() exceptions.AttributeError: 'NoneType' object has no attribute 'transport' Anywhere there are that many dots in a single expression, there's bound to be trouble. :) It's probably something rather simple, like the connection being lost near when the response ends. It's probably also pretty easy to fix (by being more careful in how lost connections are handled). If you can create a short example which reproduces the problem, that'll go a long way towards helping it get fixed. I suggest instrumenting the codepath that sets the "channel" attribute to None and watching to see what happens. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] redefine signal handlers
Jean-Paul Calderone wrote: On Wed, 4 Feb 2009 10:32:19 +, Reza Lotun wrote: Hi Gabriel, On Wed, Feb 4, 2009 at 10:15 AM, Gabriel Rossetti wrote: Hello everyone, I would like to run some cleanup code when my Twisted app receives a signal (SIGINT/SIGBREAK/SIGTERM). I saw that "_SignalReactorMixin" sets the handlers and that "ReactorBase" defines the default handlers : ... My question is how can I redefine them, other than monkey patching or inheriting the reactor and over-riding them (which I'd rather not do since some of my code uses the windows reactor when on windows since I was having problems with windows event)? Is there such a mechanism, something like setDefaultSig("SIGINT, mySigIntHandler)? One easy way to do it is to reactor.run(installSignalHandlers=False) and then manually install your own signal handlers in the usual way (using the python signal library). If you do this, you'll break spawnProcess. Fortunately, if you just install a SIGINT handler, Twisted won't stomp on it: exar...@charm:~$ python Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52)[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> def f(*a): ... print 'sigint' ...>>> import signal >>> signal.signal(signal.SIGINT, f) >>> from twisted.internet import reactor >>> reactor.run() sigint sigint sigint Quit exar...@charm:~$ Jean-Paul Ok, this is what I have been doing, but I suspect that it is the reason that my reactor won't stop sometimes (when SIGINT is sent). It usually happens with code using threads or wxpython support. I was wondering if my redefining handlers wasn't messing things up somehow... Gabriel ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] installation help
On Wed, 4 Feb 2009 13:35:59 +, Arie Lakeman wrote: Hi, I've taken an interest in event driven programming, I came across twisted and liked the look of it, I've tried downloading the .dmg from twistedmatrix/trac unfortunately after trying multiple installations including direct installation to both the folder where python is found (under system/frameworks...) and by changing the sys.path to another directory I fail to be able to import twisted in the interactive prompt. I have a hunch I may be missing OpenSSL (occassional errors) but I don't know enough about this (after googling) to be able to resolve this. I've never tried to install Twisted on OS X except by using setup.py to do a single-user installation. However, I'm pretty sure that if you provide more details about how exactly the installation process fails you'll be more likely to get someone else who does have a clue to help you out. Be detailed in what steps you took and what the consequences were. Where there are errors, include them verbatim. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] redefine signal handlers
On Wed, 4 Feb 2009 10:32:19 +, Reza Lotun wrote: Hi Gabriel, On Wed, Feb 4, 2009 at 10:15 AM, Gabriel Rossetti wrote: Hello everyone, I would like to run some cleanup code when my Twisted app receives a signal (SIGINT/SIGBREAK/SIGTERM). I saw that "_SignalReactorMixin" sets the handlers and that "ReactorBase" defines the default handlers : ... My question is how can I redefine them, other than monkey patching or inheriting the reactor and over-riding them (which I'd rather not do since some of my code uses the windows reactor when on windows since I was having problems with windows event)? Is there such a mechanism, something like setDefaultSig("SIGINT, mySigIntHandler)? One easy way to do it is to reactor.run(installSignalHandlers=False) and then manually install your own signal handlers in the usual way (using the python signal library). If you do this, you'll break spawnProcess. Fortunately, if you just install a SIGINT handler, Twisted won't stomp on it: exar...@charm:~$ python Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52) [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> def f(*a): ... print 'sigint' ... >>> import signal >>> signal.signal(signal.SIGINT, f) >>> from twisted.internet import reactor >>> reactor.run() sigint sigint sigint Quit exar...@charm:~$ Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted Conch SSH timeout
Could you give me an example of how handling the connectionLost() could be accomplished? I'm not sure where I need to integrate that into what I've already got. -Ray On Mon, Feb 2, 2009 at 7:32 PM, Paul Swartz wrote: > On Mon, Feb 2, 2009 at 2:22 PM, ray terrill > wrote: > > I've got a Twisted Conch/SSH script that I inherited which I'm using to > run > > commands remotely on some servers. I'd like to add a timeout for servers > > that are down, etc (the script currently hangs on these), but I'm not > sure > > where to add that functionality. > > It's probably hanging because you're not handling connectionLost() in > the Channel or the Transport. One of those should get called when the > connection stops (including via TCP timeout). > > -p > -- > Paul Swartz > paulswartz at gmail dot com > http://paulswartz.net/ > AIM: z3penguin > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] RE: [Stackless] Maximum Recursion Depth Error with Twisted 8.2/Stackless 2.6.1
Hi Kristjan: Thanks for the explanation and the effort. Tomorrow, I'll hunt down a few things on my end. If you find ToyProcessor5 too convoluted, I'll post the code for another prototype that should be easier to follow. Cheers, Andrew --- On Tue, 2/3/09, Kristján Valur Jónsson wrote: > From: Kristján Valur Jónsson > Subject: RE: [Stackless] Maximum Recursion Depth Error with Twisted > 8.2/Stackless 2.6.1 > To: "Andrew Francis" , > "twisted-python@twistedmatrix.com" > Cc: "stackl...@stackless.com" > Date: Tuesday, February 3, 2009, 1:49 AM > Stackless does stack copying if it senses that the C stack > has grown to deep. But it still tries to maintain the > recursion depth limit set in "sys". There is > probably a bug somewhere in the bookkeeping of this. > I'll try trawling through the code for the > corresponding stuff If I find the time today. > K > > -Original Message- > From: stackless-boun...@stackless.com > [mailto:stackless-boun...@stackless.com] On Behalf Of Andrew > Francis > Sent: 2. febrúar 2009 22:48 > To: twisted-python@twistedmatrix.com > Cc: stackl...@stackless.com > Subject: [Stackless] Maximum Recursion Depth Error with > Twisted 8.2/Stackless 2.6.1 > > Hi Colleagues: > > I recently updated to Stackless 2.6.1 and Twisted 8.2. I > executed the following programme that works fine under > Stackless 2.5.2 and Twisted (including 8.2) and I received > the following error: > > traceback (most recent call last): > File "ToyProcessor5.py", line 60, in > twistedReactor > reactor.run() > File > "/usr/local/lib/python2.6/site-packages/twisted/internet/base.py" > , line 1048, in run > self.mainLoop() > --- --- > File > "/usr/local/lib/python2.6/site-packages/twisted/internet/base.py" > , line 1057, in mainLoop > self.runUntilCurrent() > File > "/usr/local/lib/python2.6/site-packages/twisted/internet/base.py" > , line 707, in runUntilCurrent > log.deferr() > File > "/usr/local/lib/python2.6/site-packages/twisted/python/log.py", > l ine 153, in err > _stuff = failure.Failure() > File > "/usr/local/lib/python2.6/site-packages/twisted/python/failure.py > ", line 265, in __init__ > parentCs = reflect.allYourBase(self.type) > ", line 542, in allYourBase > accumulateBases(classObj, l, baseClass) > File > "/usr/local/lib/python2.6/site-packages/twisted/python/reflect.py", > line 550, in accumulateBases > accumulateBases(base, l, baseClass) > File > "/usr/local/lib/python2.6/site-packages/twisted/python/reflect.py", > line 550, in accumulateBases > accumulateBases(base, l, baseClass) > File > "/usr/local/lib/python2.6/site-packages/twisted/python/reflect.py", > line 550, in accumulateBases > accumulateBases(base, l, baseClass) > File > "/usr/local/lib/python2.6/site-packages/twisted/python/reflect.py", > line 550, in accumulateBases > accumulateBases(base, l, baseClass) > exceptions.RuntimeError: maximum recursion depth > exceeded > > Similarily written programmes fail in the same fashion. > What is suspicious is that there is no recursion in the > offending section. Even if I create one worker tasklet, I > get the same error. > > def twistedReactor(): > l = task.LoopingCall(stackless.schedule) > l.start(.01) > reactor.run() > > however, if I change > > l = task.LoopingCall(stackless.schedule) > > to > > l = task.LoopingCall(tick) > > and tick is > > def tick(): > stackless.schedule() > > the programme works. Although a work around, I would like > to find the real problem. > > What particular worries me is when I created small test > examples, I was not able to recreate the problem. Something > else is going on > > What I would appreciate is some hints as to what may be > happening. > > - I normally don't expect "Maximum Recursion > Depth" errors in Stackless. > - What is the reflect.allYourBases stuff? > - What is log.deferr? (I don't recall seeing that > method). > > If I get few clues, it would make it easier for me to write > new tests and zero in on the problem. Hopefully the problem > is with my code rather than Stackless 2.6.1 and/or Twisted > 8.2 > > I have included some sample code. Unfortunately > ToyProcessor5.py is a bit large (I have newer code that is > smaller but requires more files) > > Cheers, > Andrew ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] installation help
Hi, Firstly, I either use the built-in version of Idle on mac which is 2.5, or I'm using ActivePython 2.6, I'm not familiar with binaries so to save time this is what I used. I then install the .dmg of twisted to a file I create on my desktop, then go into sys.path and add /Users/username/Desktop, then trying 'import twisted' I get the following error: Traceback (most recent call last): File "", line 1, in import twisted File "/Users/arielakeman/Desktop/twisted/__init__.py", line 11, in import rand, crypto, SSL, tsafe ImportError: No module named OpenSSL.crypto I looked for but could not find an install.py file, and reading the readme which accompanies twisted under installation it simply says: 'see INSTALL'. I'm running Mac OS 10.5.6 and would massively appreciate any help getting twisted working, be it binaries or dmg! Thanks Arie On Wed, Feb 4, 2009 at 2:56 PM, Jean-Paul Calderone wrote: > On Wed, 4 Feb 2009 13:35:59 +, Arie Lakeman > wrote: > >> Hi, >> I've taken an interest in event driven programming, I came across twisted >> and liked the look of it, I've tried downloading the .dmg from >> twistedmatrix/trac unfortunately after trying multiple installations >> including direct installation to both the folder where python is found >> (under system/frameworks...) and by changing the sys.path to another >> directory I fail to be able to import twisted in the interactive prompt. >> I >> have a hunch I may be missing OpenSSL (occassional errors) but I don't >> know >> enough about this (after googling) to be able to resolve this. >> > > I've never tried to install Twisted on OS X except by using setup.py to do > a single-user installation. However, I'm pretty sure that if you provide > more details about how exactly the installation process fails you'll be > more likely to get someone else who does have a clue to help you out. > > Be detailed in what steps you took and what the consequences were. Where > there are errors, include them verbatim. > > Jean-Paul > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] most efficient new connection rate limiting?
We need to limit the new connection rate from a given IP, such that for example 10.10.1.10 cannot connect more often than 10 times per minute. This is a high-volume TCP Twisted server, with about 500 new distinct IP connections per second in the normal state (and we run 8 of these on each 8 core server, so ~4000 new conns/sec per box). I am trying to find the least CPU intensive approach for this. 1) Create an dictionary {ip1:count1, ip2: count2, .} in the server, and check the counts for each incoming connection. Disconnect with transport.loseConnection() if the threshold for ip:count is exceeded. Reset this dictionary to empty dict {} every minute with reactor.callLater timer. 2) Use some Twisted rate limiter API that I am not familiar with? 3) Use iptables rate-limiting module like so: iptables -I INPUT -p tcp --dport -i eth0 -m state --state NEW -m recent --set iptables -I INPUT -p tcp --dport -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 10 -j DROP Which one of these approaches would you recommend? ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted Conch SSH timeout
On Wed, Feb 4, 2009 at 11:49 AM, ray terrill wrote: > Could you give me an example of how handling the connectionLost() could be > accomplished? I'm not sure where I need to integrate that into what I've > already got. For example: class ClientCommandTransport( transport.SSHClientTransport): def __init__(self, username, password, command): self.username = username self.password = password self.command = command def verifyHostKey(self, pubKey, fingerprint): # in a real app, you should verify that the fingerprint matches # the one you expected to get from this server return defer.succeed(True) def connectionSecure(self): self.requestService( PasswordAuth(self.username, self.password, ClientConnection(self.command))) def connectionLost(self, reason): print 'do something because the connection went away' HTH, -p -- Paul Swartz paulswartz at gmail dot com http://paulswartz.net/ AIM: z3penguin ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] most efficient new connection rate limiting?
On 2009-02-04, Alec Matusis wrote: > 1) Create an dictionary {ip1:count1, ip2: count2, .} in the server, and > check the counts for each incoming connection. > Disconnect with transport.loseConnection() if the threshold for ip:count is > exceeded. > Reset this dictionary to empty dict {} every minute with reactor.callLater > timer. > > 2) Use some Twisted rate limiter API that I am not familiar with? > > 3) Use iptables rate-limiting module like so: > iptables -I INPUT -p tcp --dport -i eth0 -m state --state NEW -m recent > --set > iptables -I INPUT -p tcp --dport -i eth0 -m state --state NEW -m recent > --update --seconds 60 --hitcount 10 -j DROP I would recommend approach (3) because approach (1) will do an accept() of the connection and then drop it, giving the host on the other end a syn/ack transaction followed by a closed connection, and then it will probably attempt to reconnect immediately. Whereas (3) relies on the tcp/ip stack of the connecting host to send a bunch of syn packets until it gets through, or times out and follows its timeout logic. -- Regards, Stephen Thorne Development Engineer NetBox Blue - 1300 737 060 NetBox Blue is proud to be a sponsor and exhibitor at IBM's Solutions Showcase 2009 events. These are held in Perth, Adelaide, Brisbane, Sydney and Melbourne in February and March. For more details and to register please visit: http://www.ibm.com/solutionsshowcase/au Scanned by the NetBox from NetBox Blue (http://netboxblue.com/) ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python