[Twisted-Python] getPeer() inside conch

[Benjamin Rutt]

Hello.  I'm moderately experienced with twisted, and am starting out writing
an ssh server using conch.  I'm writing a protocol which inherits from the
recvline.HistoricRecvLine class. It has connectionMade() and lineReceived()
methods so it feels a lot like the twisted.protocols.basic.LineReceiver
class I'm used to using.

But, it seems the recvline.HistoricRecvLine class wants me to use
self.terminal rather than self.transport to talk to the client.  That's
fine, but it seems that self.terminal lacks a getPeer() method.  How do I
find out the IP address of the client inside a conch ssh server?  (For
starters, I wish to log all successful client connections so I know who's
connecting).

I searched and found the below issue.  I assume it is related, but could
someone confirm?

http://twistedmatrix.com/trac/ticket/2453

Thanks.
-- 
Benjamin Rutt
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] getPeer() inside conch

[Benjamin Rutt]

The code:

print 'hello, peer %s' % (self.terminal.transport.getPeer())

Fails with:

AttributeError: SSHSessionProcessProtocol instance has no attribute
'getPeer'

One possible patch to twisted 8.2.0's conch/ssh/session.py's
SSHSessionProcessProtocol class to add a getPeer() method would be:

$ diff -u session.py.orig session.py
--- session.py.orig 2009-06-10 06:30:33.0 -0400
+++ session.py  2009-09-29 12:30:01.622444000 -0400
@@ -13,7 +13,7 @@

 import struct

-from twisted.internet import protocol
+from twisted.internet import protocol, address
 from twisted.python import log
 from twisted.conch.interfaces import ISession
 from twisted.conch.ssh import common, channel
@@ -210,6 +210,10 @@
 def loseConnection(self):
 self.session.loseConnection()

+def getPeer(self):
+client = self.session.conn.transport.transport.client
+return address.IPv4Address('TCP', client[0], client[1])
+
 class SSHSessionClient(protocol.Protocol):

 def dataReceived(self, data):

Could someone comment on this?  (I'm fairly certain ssh runs over TCP, but
less certain that the mouthful self.session.conn.transport.transport.client
isn't invading layers of abstraction in an unfriendly way).  Thanks.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] getPeer() inside conch

[Benjamin Rutt]

(A/B) it looks like the SSHChannel implementation of getPeer() is faulty.
If I change my two-line implementation of getPeer() inside
SSHSessionProcessProtocol to simply 'return self.session.getPeer()', I get:

  File
"/home/ruttbe/dev/python/twisted-8.2.0-inst/lib/python/twisted/conch/ssh/session.py",
line 216, in getPeer
return self.session.getPeer()
  File
"/home/ruttbe/dev/python/twisted-8.2.0-inst/lib/python/twisted/conch/ssh/channel.py",
line 261, in getPeer
return('SSH', )+self.conn.transport.getPeer()
AttributeError: SSHServerTransport instance has no attribute 'getPeer'

(C) regarding adding a test, which file do you think would be suitable
here?  something in the twisted/test dir?
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] complete producer/consumer example

[Benjamin Rutt]

How does the below example look as a complete producer/consumer example?  If
it's well received, perhaps we can add it to the online documentation at
http://twistedmatrix.com/documents/current/core/howto/producers.html in the
"Further Reading" section?  I always felt that the producer/consumer (a.k.a.
high volume streaming) docs lacked a real example that users could download
and run.

#!/sw/external/python-2.6.1/bin/python
"""Serve as a sample implementation of a twisted producer/consumer
system, with a simple TCP server which asks the user how many random
integers they want, and it sends the result set back to the user, one
result per line."""

import random

from zope.interface import implements
from twisted.internet import interfaces, reactor
from twisted.internet.protocol import Factory
from twisted.protocols.basic import LineReceiver

class Producer:
"""Send back the requested number of random integers to the client."""
implements(interfaces.IPushProducer)
def __init__(self, proto, cnt):
self._proto = proto
self._goal = cnt
self._produced = 0
self._paused = False
def pauseProducing(self):
"""When we've produced data too fast, pauseProducing() will be
called (reentrantly from within resumeProducing's transport.write
method, most likely), so set a flag that causes production to pause
temporarily."""
self._paused = True
print('pausing connection from %s' %
(self._proto.transport.getPeer()))
def resumeProducing(self):
self._paused = False
while not self._paused and self._produced < self._goal:
next_int = random.randint(0, 1)
self._proto.transport.write('%d\r\n' % (next_int))
self._produced += 1
if self._produced == self._goal:
self._proto.transport.unregisterProducer()
self._proto.transport.loseConnection()
def stopProducing(self):
pass

class ServeRandom(LineReceiver):
"""Serve up random data."""
def connectionMade(self):
print('connection made from %s' % (self.transport.getPeer()))
self.transport.write('how many random integers do you want?\r\n')
def lineReceived(self, line):
cnt = int(line.strip())
producer = Producer(self, cnt)
self.transport.registerProducer(producer, True)
producer.resumeProducing()
def connectionLost(self, reason):
print('connection lost from %s' % (self.transport.getPeer()))
factory = Factory()
factory.protocol = ServeRandom
reactor.listenTCP(1234, factory)
print('listening on 1234...')
reactor.run()

Use on the client:

$ telnet localhost 1234
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
how many random integers do you want?
5
431
7201
3289
9604
6659
Connection closed by foreign host.
$

Use on the server (observe how the server pauses production sometimes - this
happens when a large data set is requested by the client):

$ ./streaming.py
listening on 1234...
connection made from IPv4Address(TCP, '127.0.0.1', 54859)
connection lost from IPv4Address(TCP, '127.0.0.1', 54859)
connection made from IPv4Address(TCP, '127.0.0.1', 54864)
pausing connection from IPv4Address(TCP, '127.0.0.1', 54864)
pausing connection from IPv4Address(TCP, '127.0.0.1', 54864)
pausing connection from IPv4Address(TCP, '127.0.0.1', 54864)
pausing connection from IPv4Address(TCP, '127.0.0.1', 54864)
connection lost from IPv4Address(TCP, '127.0.0.1', 54864)
[...]

Thanks.
-- 
Benjamin Rutt
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] complete producer/consumer example

[Benjamin Rutt]

Ok, this is done.  See http://twistedmatrix.com/trac/ticket/4171.

I decided to start by just adding the example.  For me to rework the whole
howto around this example, I'm afraid I'd need to understand how it all
works a bit more than I do now.  So, baby steps first.

Do you need anything more from me here to pick this up and add it to the
core docs?  (I wasn't sure who to assign the ticket to, and it ended up
assigning it to glyph).

Thanks.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] waiting on any deferred in a list?

[Benjamin Rutt]

Hi all,

I've been using a pattern for quite some time to collect a bunch o'
deferreds in a list and then yield on them one at a time.  Here is some
untested pseudocode:

---
@defer.inlineCallbacks
def foo( ls ):
defs = []

# get all the deferreds going
for item in ls:
d = some_deferred_returning_function( item )
defs.append( ( item, d ) ) # associate item with deferred

# process results not as they come in but in the original order
for ( item, d ) in defs:
result = yield d
print 'The result on item', item, 'is', result
---

In this way, I can get good concurrency (by spinning up all deferreds at
once) but also I can iterate on the results using a straightforward
traversal of the list data structure and yield.  This is working great for
me.  But on occasion I'd prefer that I can access the results not in the
order that I originally got, but rather in the order of completion of the
deferreds, so the faster ones come in first, and the slower ones don't hold
up faster ones.  The usual case when I'd want this is when I have "slower"
deferreds near the front, and "faster" deferreds near the back.  I can't
help but think that there's some async data structure out there for me that
can solve this.  I've used DeferredList before, and I can't see that it
does what I want:  it gives you a choice between getting all results only
when they are all done (default behavior), or getting the first one that's
ready (fireOnOneCallback=True).  I want all results, but I want the
processing of the results to unfold as the results come in.

Thanks for any ideas.
-- 
Benjamin Rutt
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] integrating a python REPL into my existing twisted application

[Benjamin Rutt]

Dear twisted-python,

I’m trying to integrate a python REPL into my existing twisted 8.2.0
client-side program (which talks to another twisted server running a custom
protocol).  This client side program uses twisted.internet.stdio.StandardIO
already to interact with the user, to read commands from the user and send
them to the server.  What’s the best way to add functionality to drop into a
python REPL?  I adapted the doc/core/examples/stdin.py code below to drop
into the ‘code’ stdlib module to try to model for the mailing list what my
larger client program is trying to do.  It works fine on linux, until you
enter a command which outputs a large amount of data, at which point you get
a “Resource temporarily unavailable” as can be seen below.  I’m assuming the
issue is related to StandardIO taking control of reads/writes to the stdio
file descriptors, and not integrating well with python code that writes
directly to sys.stderr, but I’m not sure what the right solution is.  I
don’t mind that the call to ‘code.interact()’ blocks the reactor for my use
case, but I don’t mind if better solutions are available which don’t have
that problem, either.

Modified doc/core/examples/stdin.py:
---
"""
An example of reading a line at a time from standard input
without blocking the reactor.
"""

from twisted.internet import stdio
from twisted.protocols import basic

class Echo(basic.LineReceiver):
from os import linesep as delimiter

def connectionMade(self):
self.transport.write('> ')

def lineReceived(self, line):
self.sendLine('Echo: ' + line)
self.transport.write('> ')
import code
code.interact()

def main():
stdio.StandardIO(Echo())
from twisted.internet import reactor
reactor.run()

if __name__ == '__main__':
main()
---
A sample run:

$ ./twisted-stdio.py
> foo
Python 2.6.1 (r261:67515, Jun  4 2009, 11:06:31)
[GCC 4.1.2 20071124 (Red Hat 4.1.2-43)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> 11 * 'x'
'xxx'
>>> 1 * 'x'
'

Re: [Twisted-Python] integrating a python REPL into my existing twisted application

[Benjamin Rutt]

On Thu, May 26, 2011 at 9:38 AM,  wrote:

>
> In addition to what Andrew said, I'll also point out that rather than
> making code that expects blocking writes to stdout work by putting
> stdout back into blocking mode, you can make it work by instead
> providing a stdout which will buffer anything that cannot immediately be
> written non-blockingly.  StandardIO gives you exactly this; its `write`
> knows how to deal with non-blocking descriptors and buffer as necessary.
> This is why manhole doesn't have the same problem as your code calling
> directly into `code.interact`.  For details,
>  >.
>
>
Thanks.  I'm having trouble integrating this or Andrew's idea into my
example.  The simplest case doesn't quite work right for me (I see '>>>'
prompts, and can interact with the python interpreter but the banner appears
at the end of the REPL after I press Ctrl+D, and then I never see the '>'
prompts from my original program anymore)...was this what you had in mind?
See the diff below.

@@ -6,6 +6,7 @@
 without blocking the reactor.
 """

+import sys
 from twisted.internet import stdio
 from twisted.protocols import basic

@@ -22,7 +23,8 @@
 code.interact()

 def main():
-stdio.StandardIO(Echo())
+stdio_obj = stdio.StandardIO(Echo())
+sys.stderr = stdio_obj
 from twisted.internet import reactor
 reactor.run()


-- 
Benjamin
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] twisted and kerberos

[Benjamin Rutt]

Hello twisted experts,

We are trying to add kerberos authentication to our existing twisted server.
 We haven't yet found a twisted specific library that can do this; any
pointers?

What we have found is python-krbV (https://fedorahosted.org/python-krbV/),
but we end up using it along with code like this to take over the socket
ourselves from twisted, when the client connects:

self.transport.stopReading()
self.transport.stopWriting()
self.transport.socket.setblocking(1)

and then we use the self.transport.socket directly in blocking mode in
another thread (so it won't block the reactor).  In particular, we use code
like this to start the authentication process:

ac = krbV.default_context().sendauth(self.transport.socket, ...)

…once the client has authenticated, we use:

self.transport.socket.setblocking(0)
self.transport.startReading()
self.transport.startWriting()

to let twisted take over the socket again for the rest of the conversation.

So in the event there are no ready-to-go twisted libraries for kerberos, is
the above approach totally insane?
-- 
Benjamin Rutt
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] how to ensure writes to stdout via stdio.StandardIO are completed before we shutdown the reactor?

[Benjamin Rutt]

Hi twisted community,

What’s the best way to ensure writes to stdout via stdio.StandardIO
are completed before we shutdown the reactor?   The use case is to
emit an error message before we shutdown the reactor and stop our
program.  Dummy code snippet:


#!/usr/bin/python

from twisted.internet import reactor, stdio
from twisted.protocols.basic  import LineReceiver

class StdinInteractor(LineReceiver):
  delimiter = '\n'

def go():
  stdio_proto = StdinInteractor()
  stdio.StandardIO(stdio_proto)
  stdio_proto.transport.write('did we get this message?\n')
  if 0:
    reactor.stop()
  else:
    reactor.callLater(1.0, reactor.stop)

reactor.callWhenRunning(go)
reactor.run()


If I change ‘if 0’ to ‘if 1’, I don’t get the message on stdout.  The
‘reactor.callLater(1.0, reactor.stop)’ is obviously a hack to let the
reactor send out the buffered message before shutting down.  Any
ideas?

Thanks,

-- 
Benjamin Rutt

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] how to ensure writes to stdout via stdio.StandardIO are completed before we shutdown the reactor?

[Benjamin Rutt]

That worked so well, that I was speechless for awhile.  Thanks!

On Wed, Feb 1, 2012 at 1:01 AM,   wrote:
> On 03:59 am, rut...@osu.edu wrote:
>>Hi twisted community,
>>
>>What’s the best way to ensure writes to stdout via stdio.StandardIO
>>are completed before we shutdown the reactor?   The use case is to
>>emit an error message before we shutdown the reactor and stop our
>>program.  Dummy code snippet:
>
> If you call the transport's loseConnection method, then the protocol's
> connectionLost method will be called after the output buffer is
> completely written.
>
> Jean-Paul
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python



-- 
Benjamin Rutt

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] MemoryError in twisted causes the program to exit, or not

[Benjamin Rutt]

Hi, I am trying to understand what will happen in my long-running twisted
server program when the available memory is low.

If I run the following program with increasing numeric seeds as arguments,
0, 1, 2, 3, 4, … , some of the time the out of memory condition will crash
and exit the program right away (when the MemoryError is raised inside
twisted core code), and other times it will just carry on with an
UnhandledError (if the MemoryError is raised inside my hi() function below).

Sample code:

---
#!/usr/bin/env python

import sys, resource, random, traceback

from twisted.internet import reactor

resource.setrlimit(resource.RLIMIT_AS, (180 * 1024 * 1024,
resource.RLIM_INFINITY))

data = []
random.seed(int(sys.argv[1]))

def hi():
data.append('x' * random.randint(1, 1))
reactor.callLater(0, hi)

def main():
reactor.callLater(0, hi)

reactor.callWhenRunning(main)
reactor.run()
 ---

I am using python 2.7.1, twisted 11.0.0.

Sample runs follow:

bash$ n=0; while true; do echo "starting rep $n"; ./twisted-oom.py $n;
n=$(($n+1)); echo; done
starting rep 0
Traceback (most recent call last):
  File “./twisted-oom.py", line 21, in 
reactor.run()
  File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 1162, in run
self.mainLoop()
  File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 1177, in mainLoop
log.err()
  File "/sw/external/twisted-py27-11.0.0/lib/python/twisted/python/log.py",
line 197, in err
_stuff = failure.Failure()
MemoryError

starting rep 1
Traceback (most recent call last):
  File ". /twisted-oom.py", line 21, in 
reactor.run()
  File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 1162, in run
self.mainLoop()
  File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 1177, in mainLoop
log.err()
MemoryError

starting rep 2
Unhandled Error
Traceback (most recent call last):
  File "./twisted-oom.py", line 21, in 
reactor.run()
  File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 1162, in run
self.mainLoop()
  File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 1171, in mainLoop
self.runUntilCurrent()
---  ---
  File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 793, in runUntilCurrent
call.func(*call.args, **call.kw)
  File "/home/ruttbe/dev/oneoff/twisted-oom.py", line 14, in hi
data.append('x' * random.randint(1, 1))
exceptions.MemoryError:

On the first two runs, the program exited quickly, but on the last rep
("starting rep 2"), it just carried on running with the reactor still
running.  It would be nice if there was a way to make the out of memory
behavior consistent, so I would know that it would either always carry on
running with a MemoryError stack trace, or always crash out and end the
program.  Maybe runUntilCurrent should not catch MemoryError from user code
and should let it propagate out?  Also wondering if there are other
different behaviors in other areas of the code (such as callbacks into user
code such as dataReceived, outReceived, lineReceived, etc.).  i.e. is the
observed behavior (that MemoryError in user code is caught and MemoryError
in twisted core is not) intentional, or an accident?

Thanks,
-- 
Benjamin Rutt
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] inlineCallbacks loses custom exception type?

[Benjamin Rutt]

Hi,

I am hitting an issue with inlineCallbacks-decorated functions.  It seems
that from within an inlineCallbacks-decorated func, if you yield on another
deferred-returning func, and that deferred-returning func returns a
deferred which will errback with a Failure wrapping a user defined
exception type, then the user defined exception gets repackaged as a common
Exception by the time you are in the inlineCallbacks-decorated function.
It seems to be related to the fact that inside an inlineCallbacks-decorated
func, if you yield on a func, and that func returns a deferred which errs
back, twisted will convert the errback into a synchronous exception that it
raises for you.  That’s all a bit wordy, so maybe some code will help
clarify:

$ cat inlinecallbacks-loses-exception-types.py
#!/usr/bin/env python

import sys, traceback
from twisted.internet import defer, reactor

class MyException(Exception):
  pass

def go2():
"An ordinary deferred-returning function that will return a deferred
which is ready to errback()"
try:
raise MyException('xyz')
except Exception as e:
return defer.fail() # errs back with current exception context

return defer.succeed(1) # unreachable code

@defer.inlineCallbacks
def go_inlinecb():
try:
x = yield go2()
print 'no exception raised'
except MyException as e:
print 'got my custom exception'
traceback.print_exc()
except Exception as e:
print 'got a garden variety exception'
traceback.print_exc()

def go_noinlinecb():
d = go2()
def cb(data):
print 'got data in cb: %s' % (data)
def eb(failure):
try:
failure.raiseException()
except MyException as e:
print 'got my custom exception'
traceback.print_exc()
except Exception as e:
print 'got a garden variety exception'
traceback.print_exc()
d.addCallbacks(cb, eb)

if sys.argv[1] == 'inline':
reactor.callWhenRunning(go_inlinecb)
elif sys.argv[1] == 'noinline':
reactor.callWhenRunning(go_noinlinecb)
reactor.callLater(2, reactor.stop)
reactor.run()

$ ./inlinecallbacks-loses-exception-types.py inline
got a garden variety exception
Traceback (most recent call last):
  File "./inlinecallbacks-loses-exception-types.py", line 21, in go_inlinecb
x = yield go2()
Exception: xyz

$ ./inlinecallbacks-loses-exception-types.py noinline
got my custom exception
Traceback (most recent call last):
  File "./inlinecallbacks-loses-exception-types.py", line 36, in eb
failure.raiseException()
  File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/python/failure.py",
line 338, in raiseException
raise self.type, self.value, self.tb
MyException: xyz

So as you can see, it seems that if you use inlineCallbacks, you cannot
‘except MyException’ and catch your own custom exceptions.  Somehow a
generic Exception is constructed.  Is there any workaround or is this a
known issue?

Thanks,

-- 
Benjamin Rutt
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] inlineCallbacks loses custom exception type?

[Benjamin Rutt]

That's a useful data point, thanks.  I usually don't use unreleased
revisions of twisted, but I'll try the latest release for starters and see
what the result is.  Forgot to mention before, I am on python 2.7.1 and
twisted 11.0.0, so there's a chance this is fixed in a later release from
what I am running.

On Fri, Dec 7, 2012 at 10:00 AM,  wrote:

> On 6 Dec, 03:36 am, rut...@osu.edu wrote:
> >
> >$ ./inlinecallbacks-loses-exception-types.py inline
> >got a garden variety exception
> >Traceback (most recent call last):
> >  File "./inlinecallbacks-loses-exception-types.py", line 21, in
> >go_inlinecb
> >x = yield go2()
> >Exception: xyz
>
> I'm unable to reproduce this error behavior using Twisted trunk@HEAD on
> Ubuntu 12.04.
>
> Jean-Paul
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
>


-- 
Benjamin Rutt
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] deferToThread and thrown exceptions break gc - ticket 3853

[Benjamin Rutt]

Hi,

I believe I hit http://twistedmatrix.com/trac/ticket/3853 in production
this week, in the form of what looked like a resource leak, but turned out
to be a case of a resource held for much longer than expected.  (I'm using
python 2.7, and twisted 12.3.0).

That is, I had a function invoked by deferToThread, that searched a file by
memory mapping it using the 'mmap' module.  And if my function could not
find what it was looking for in the file, it raised an exception at the
end.  I observed that my process was still hanging on to the mmap, due to
my local variable representing the mmap (which I presumed would have been
gc'd as it went out of scope as part of the throw) not being gc'd when the
function ended.  I can work around it in my function by setting that local
variable to None right before I throw, but this is python, not C, and in
python automatic memory management is assumed.

So, +1 for that issue getting fixed.   From the issue it looks like there
is a deeper issue in python itself that could fix this, but as the fix
discussion is moving at a glacial pace, I say +1 for a twisted workaround,
which from the trac discussion, sounds possible.  Thanks,
-- 
Benjamin
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] deferToThread and thrown exceptions break gc - ticket 3853

[Benjamin Rutt]

I am not sure that that trac issue contains a patch that will fix it.  The
most recent patch on that trac issue is for twisted 8.1, and it does not
apply cleanly to twisted 12.3.0.  When I tried to place the 2 lines that
form the patch in the most obvious place in the 12.3.0 codebase (before the
code 'self.waiters.append(ct)') I could not get it to fix the repro I have.
 Here's my repro:

from twisted.internet import reactor
from twisted.internet.threads import deferToThread

class SomeClass(object):
def __del__(self):
print 'SomeClass\'s destructor was called!'

def foo():
sc = SomeClass()
raise RuntimeError('bah')

def shutmedown(data):
print 'reactor shutdown happening now'
reactor.stop()

def go():
d = deferToThread(foo)
d.addCallbacks(shutmedown, shutmedown)
reactor.callWhenRunning(go)
reactor.run()

and running it:

$ ./twisted-defertothread-bug.py
reactor shutdown happening now
SomeClass's destructor was called!

The bug is present, as the instance of SomeClass only gets gc'd once the
reactor shuts down (taking the thread pool with it), not when the foo()
function terminates.  Obviously I'm using CPython, which uses reference
counting gc, so there is no reason to delay the reclaim of the SomeClass
instance until after the function exit.


On Wed, Jun 26, 2013 at 1:03 PM, Christopher Armstrong <
ra...@twistedmatrix.com> wrote:

> On Wed, Jun 26, 2013 at 9:15 AM, Benjamin Rutt  wrote:
>
>> Hi,
>>
>> I believe I hit http://twistedmatrix.com/trac/ticket/3853 in production
>> this week, in the form of what looked like a resource leak, but turned out
>> to be a case of a resource held for much longer than expected.  (I'm using
>> python 2.7, and twisted 12.3.0).
>>
>> That is, I had a function invoked by deferToThread, that searched a file
>> by memory mapping it using the 'mmap' module.  And if my function could not
>> find what it was looking for in the file, it raised an exception at the
>> end.  I observed that my process was still hanging on to the mmap, due to
>> my local variable representing the mmap (which I presumed would have been
>> gc'd as it went out of scope as part of the throw) not being gc'd when the
>> function ended.  I can work around it in my function by setting that local
>> variable to None right before I throw, but this is python, not C, and in
>> python automatic memory management is assumed.
>>
>> So, +1 for that issue getting fixed.   From the issue it looks like there
>> is a deeper issue in python itself that could fix this, but as the fix
>> discussion is moving at a glacial pace, I say +1 for a twisted workaround,
>> which from the trac discussion, sounds possible.  Thanks,
>>
>>
> Like you said, it looks like there's already a patch that can solve the
> problem. All it needs before it can be reviewed is tests. If you could
> spare some time to write one (or however many are needed) then that would
> increase the ticket's chances for resolution quite a bit :)
>
>
> --
> Christopher Armstrong
> http://radix.twistedmatrix.com/
> http://planet-if.com/
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>


-- 
Benjamin Rutt
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] twisted.names.client.getHostByName() vs. reactor.resolve()

[Benjamin Rutt]

Hi,

I need to do a DNS resolution of a hostname to an IPv4 address in order to
then send out a UDP packet via twisted’s APIs.  There appears to be 2 ways
to do this.  Is it better to use twisted.names.client.getHostByName(), or
is it better to use reactor.resolve()?  Is there a bias for one over the
other?  Is one deprecated and the other isn’t?

BTW they appear to behave differently; at work, I have a hostname that the
one (getHostByName) resolves to None in the callback, and the other
(reactor.resolve) resolves to a valid IP address in the callback.  That
analysis is based on twisted 12.3.0 on linux.  Naturally I'm preferring
reactor.resolve at this point, as it works better for the hostname in
question (it happens to be a hostname that uses a CNAME record to alias to
another hostname).

Thanks,


-- 
Benjamin Rutt
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python