[Twisted-Python] Correct pattern to do SSH forwarding in a GUI
I am using twisted in a GUI application to do port forwarding over an SSH connection. All the network connections of the application (database, etc...) are routed through this connection, to simplify server configuration and secure things. My first attempt was to run the reactor in the main thread, but the application was "freezing" when e.g. doing a SELECT with SQLAlchemy/psycopg2 (I guess because psycopg2 just waits for data and blocks the twisted reactor). So I put reactor.run() in a thread and it works, but I'm getting strange issues on some platforms (for example I can't connect at all on *some* OSX computers, getting a "reactor stopping" log from twisted without any further error), and I feel I'm doing the whole thing terribly wrong. So my question is, how to do proper SSH port forwarding with conch in an application that also uses the tunnels not the asynchronous way ? If running the reactor in a thread is OK, are there particular things to watch out for ? ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Correct pattern to do SSH forwarding in a GUI
gl...@divmod.com wrote: On 04:19 am, luper.ro...@gmail.com wrote: My first attempt was to run the reactor in the main thread, but the application was "freezing" when e.g. doing a SELECT with SQLAlchemy/psycopg2 (I guess because psycopg2 just waits for data and blocks the twisted reactor). So I put reactor.run() in a thread and it works, but I'm getting strange issues on some platforms (for example I can't connect at all on *some* OSX computers, getting a "reactor stopping" log from twisted without any further error), and I feel I'm doing the whole thing terribly wrong. These errors sound like you are making Twisted API calls from threads other than the reactor thread by accident. There's no obvious fix except for "don't do that" :). Yes, I did not pay attention to separate Twisted from the main thread at all, apart from putting reactor.run() in another thread. My suggestion would be to move the SQLAlchemy/psycopg2 into threads, with e.g. deferToThread, rather than putting the reactor into a thread. Then the reactor won't freeze while the blocking SQL stuff happens. So my question is, how to do proper SSH port forwarding with conch in an application that also uses the tunnels not the asynchronous way ? If running the reactor in a thread is OK, are there particular things to watch out for ? There's no reason that running the reactor in a thread shouldn't work; it's just generally tricky (especially if you're working with an existing codebase) to completely avoid calling Twisted APIs from other threads. This implies too much modifications in my code, I think I will put the whole Twisted stuff in a separate process and interact with it via simple IPC, as I don't use that much of Twisted in the GUI code anyway. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] KeyError when an errback is triggered within a conch TCP tunnel
Sorry I have not been able to find a better title, please see the
attached example which may be more descriptive.
Here is what it does:
* set up an SSH connection
* open a TCP forwarding channel through this connection
* do something through the tunnel that will trigger an errback (in the
example, try to do an XMLRPC call with no one listening on the
tunnel's end)
This scenario gives me the following traceback (tested with Twisted
8.1.0 and 8.2.0):
-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
File
"/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/python/log.py",
line 84, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File
"/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/python/log.py",
line 69, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File
"/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/python/context.py",
line 59, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File
"/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/python/context.py",
line 37, in callWithContext
return func(*args,**kw)
File
"/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/internet/selectreactor.py",
line 156, in _doReadOrWrite
self._disconnectSelectable(selectable, why, method=="doRead")
File
"/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/internet/posixbase.py",
line 193, in _disconnectSelectable
selectable.connectionLost(f)
File
"/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/internet/tcp.py",
line 520, in connectionLost
protocol.connectionLost(reason)
File
"/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/conch/ssh/forwarding.py",
line 129, in connectionLost
self.channel.loseConnection()
File
"/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/conch/ssh/channel.py",
line 253, in loseConnection
self.conn.sendClose(self)
File
"/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/conch/ssh/connection.py",
line 491, in sendClose
self.channelsToRemoteChannel[channel]))
exceptions.KeyError:
at 0x972048c>
-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
It works well in the other cases, i.e.:
* the errback is triggered but the tunnel is not used (comment line 57,
uncomment line 59)
* the errback is not triggered (start the XMLRPC server by uncommenting
line 100), using the tunnel or not
You get the error only when the tunnel is used AND the errback is triggered.
To run the example execute it with an user name that can login using ssh
on your host, e.g. 'python test.py someuser' and enter its password.
Cheers,
Luper
#!/usr/bin/env python
import sys
from getpass import getpass
from twisted.python import log
from twisted.conch.ssh import connection, transport, userauth, keys, \
forwarding, channel
from twisted.conch.ssh.common import NS
from twisted.internet import defer, protocol, reactor
from twisted.web import server, xmlrpc
from twisted.web.xmlrpc import Proxy
conn = None
def connect(user, host, port=22):
return reactor.connectTCP(host, port, Factory(user, Transport))
class Factory(protocol.ClientFactory):
def __init__(self, user, protocol):
self.user = user
self.protocol = protocol
def buildProtocol(self, addr):
p = self.protocol(self.user)
p.factory = self
return p
class Transport(transport.SSHClientTransport):
def __init__(self, user):
self.user = user
def verifyHostKey(self, pubkey, fingerprint):
return defer.succeed(1)
def connectionSecure(self):
self.requestService(UserAuth(self.user, Connection()))
class UserAuth(userauth.SSHUserAuthClient):
def getPassword(self, prompt=None):
return defer.succeed(getpass("password: "))
class Connection(connection.SSHConnection):
def __init__(self, *args, **kwargs):
connection.SSHConnection.__init__(self, *args, **kwargs)
self.forwarding_sockets = []
def serviceStarted(self):
# Create a tunnel from 12345 to 54321
self.forward_port(12345, 54321)
# Do something through the tunnel
p = Proxy('http://localhost:12345/')# passing through the tunnel,
# triggers the KeyError
#p = Proxy('http://localhost:54321/') # not using the tunnel, works
print "ping...",
d = p.callRemote("ping")
def stop_ok(response):
print response
conn.disconnect()
reactor.stop()
def stop_error(error):
print "error", error
conn.disconnect()
reactor.stop()
d.addCallback(stop_ok)
[Twisted-Python] KeyError when an errback is triggered within a conch TCP tunnel
(posting it a second time with the example code inlined as pipermail did
not like the attachment)
Sorry I have not been able to find a better title, please see the
attached example which may be more descriptive.
Here is what it does:
* set up an SSH connection
* open a TCP forwarding channel through this connection
* do something through the tunnel that will trigger an errback (in the
example, try to do an XMLRPC call with no one listening on the
tunnel's end)
This scenario gives me the following traceback (tested with Twisted
8.1.0 and 8.2.0):
-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
File
"/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/python/log.py",
line 84, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File
"/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/python/log.py",
line 69, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File
"/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/python/context.py",
line 59, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File
"/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/python/context.py",
line 37, in callWithContext
return func(*args,**kw)
File
"/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/internet/selectreactor.py",
line 156, in _doReadOrWrite
self._disconnectSelectable(selectable, why, method=="doRead")
File
"/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/internet/posixbase.py",
line 193, in _disconnectSelectable
selectable.connectionLost(f)
File
"/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/internet/tcp.py",
line 520, in connectionLost
protocol.connectionLost(reason)
File
"/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/conch/ssh/forwarding.py",
line 129, in connectionLost
self.channel.loseConnection()
File
"/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/conch/ssh/channel.py",
line 253, in loseConnection
self.conn.sendClose(self)
File
"/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/conch/ssh/connection.py",
line 491, in sendClose
self.channelsToRemoteChannel[channel]))
exceptions.KeyError:
at 0x972048c>
-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
It works well in the other cases, i.e.:
* the errback is triggered but the tunnel is not used (comment line 57,
uncomment line 59)
* the errback is not triggered (start the XMLRPC server by uncommenting
line 100), using the tunnel or not
You get the error only when the tunnel is used AND the errback is triggered.
To run the example execute it with an user name that can login using ssh
on your host, e.g. 'python test.py someuser' and enter its password.
Cheers,
Luper
Example code:
-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
#!/usr/bin/env python
import sys
from getpass import getpass
from twisted.python import log
from twisted.conch.ssh import connection, transport, userauth, keys, \
forwarding, channel
from twisted.conch.ssh.common import NS
from twisted.internet import defer, protocol, reactor
from twisted.web import server, xmlrpc
from twisted.web.xmlrpc import Proxy
conn = None
def connect(user, host, port=22):
return reactor.connectTCP(host, port, Factory(user, Transport))
class Factory(protocol.ClientFactory):
def __init__(self, user, protocol):
self.user = user
self.protocol = protocol
def buildProtocol(self, addr):
p = self.protocol(self.user)
p.factory = self
return p
class Transport(transport.SSHClientTransport):
def __init__(self, user):
self.user = user
def verifyHostKey(self, pubkey, fingerprint):
return defer.succeed(1)
def connectionSecure(self):
self.requestService(UserAuth(self.user, Connection()))
class UserAuth(userauth.SSHUserAuthClient):
def getPassword(self, prompt=None):
return defer.succeed(getpass("password: "))
class Connection(connection.SSHConnection):
def __init__(self, *args, **kwargs):
connection.SSHConnection.__init__(self, *args, **kwargs)
self.forwarding_sockets = []
def serviceStarted(self):
# Create a tunnel from 12345 to 54321
self.forward_port(12345, 54321)
# Do something through the tunnel
p = Proxy('http://localhost:12345/')# passing through the
tunnel,
# triggers the KeyError
#p = Proxy('http://localhost:54321/') # not using the tunnel,
works
print "ping...",
d = p.callRemote("ping")
def stop_ok(response):
print response
conn.disconnect()
r
[Twisted-Python] Debugging Twisted
I would like to investigate on the conch error I mentionned in my
previous message ("KeyError when an errback is triggered within a conch
TCP tunnel").
My problem is Twisted catches the exceptions and the debugger won't stop
on them, how could I avoid that ? Is there a "debug" reactor ?
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Debugging Twisted
Jean-Paul Calderone wrote :
On Fri, 20 Feb 2009 19:56:54 +0100, Luper Rouch
wrote:
I would like to investigate on the conch error I mentionned in my
previous message ("KeyError when an errback is triggered within a
conch TCP tunnel").
My problem is Twisted catches the exceptions and the debugger won't
stop on them, how could I avoid that ? Is there a "debug" reactor ?
Try the --debug option to twistd.
Jean-Paul
My application is a client, how can I run it with twistd ?
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Debugging Twisted
Jean-Paul Calderone a écrit :
On Fri, 20 Feb 2009 20:15:02 +0100, Luper Rouch
wrote:
Jean-Paul Calderone wrote :
On Fri, 20 Feb 2009 19:56:54 +0100, Luper Rouch
wrote:
I would like to investigate on the conch error I mentionned in my
previous message ("KeyError when an errback is triggered within a
conch TCP tunnel").
My problem is Twisted catches the exceptions and the debugger won't
stop on them, how could I avoid that ? Is there a "debug" reactor ?
Try the --debug option to twistd.
Jean-Paul
My application is a client, how can I run it with twistd ?
See http://twistedmatrix.com/projects/core/documentation/howto/tap.html
Sorry but I don't understand how to run a _client_ application using
twistd, I know how to create servers, but the code I would like to debug
only uses conch to connect to a SSH server (OpenSSH, not Twisted).
Luper
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Re: KeyError when an errback is triggered within a conch TCP tunnel
Running the example in the debugger, neither ssh_CHANNEL_OPEN() nor ssh_CHANNEL_OPEN_CONFIRMATION() method of twisted.conch.ssh.connection.SSHConnection is called, so self.channelsToRemoteChannel dict is empty when sendClose() is called, thus the error. I am not sure if I should file a bug as I'm new to Twisted, does anyone see an obvious mistake in my code ? Cheers, Luper ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Debugging Twisted
2009/2/23 Jean-Paul Calderone :
> On Mon, 23 Feb 2009 15:11:15 +0100, Luper Rouch
> wrote:
>>
>> Jean-Paul Calderone a écrit :
>>>
>>> On Fri, 20 Feb 2009 20:15:02 +0100, Luper Rouch
>>> wrote:
>>>>
>>>> Jean-Paul Calderone wrote :
>>>>>
>>>>> On Fri, 20 Feb 2009 19:56:54 +0100, Luper Rouch
>>>>> wrote:
>>>>>>
>>>>>> I would like to investigate on the conch error I mentionned in my
>>>>>> previous message ("KeyError when an errback is triggered within a conch
>>>>>> TCP
>>>>>> tunnel").
>>>>>>
>>>>>> My problem is Twisted catches the exceptions and the debugger won't
>>>>>> stop on them, how could I avoid that ? Is there a "debug" reactor ?
>>>>>
>>>>> Try the --debug option to twistd.
>>>>>
>>>>> Jean-Paul
>>>>
>>>> My application is a client, how can I run it with twistd ?
>>>
>>> See http://twistedmatrix.com/projects/core/documentation/howto/tap.html
>>
>> Sorry but I don't understand how to run a _client_ application using
>> twistd, I know how to create servers, but the code I would like to debug
>> only uses conch to connect to a SSH server (OpenSSH, not Twisted).
>
> There is no significant difference between servers and clients in this
> respect. If you know how to create a server, you know how to create a
> client - just use the client APIs instead of the server APIs.
>
> Jean-Paul
>
Sorry I had missed TCPClient in the documentation, it works now thanks.
Luper
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Using sqlalchemy in twisted.
I would recommend sandboxing Twisted code in another process when used in conjunction with synchronous code (sqlalchemy, GUI, etc...). This proved quite simple in my project, using the multiprocessing module [1], and avoided me lots of headaches since. This way I don't mix coding styles everywhere using deferToThread(), nor do I care about threads safety madness. Now I'm not sure if this could apply to your situation, but if like me you're just routing the SA traffic through Twisted (just guessing as you mentioned deferToThread()), have not so much interaction between Twisted and the synchronous code part, it might be a better solution than writing wrappers / limiting your usage of the synchronous code. Luper [1]http://code.google.com/p/python-multiprocessing/ (now part of python 2.6) ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Using sqlalchemy in twisted.
2009/3/12 Peter Cai : > Hi, > I've read some mails in this list that recommend use sqlalchemy in separate > processes. > But if u use multiprocessing which provides only synchronous API, doesn't > you have to use deferToThread also? > Another way would be using "twisted.internet.protocol.ProcessProtocol", > which use pipes. But can we pass SA Objects through pipes? > If you can write a little code to describe how you use multiprocessing, I > will be very thankful! multiprocessing [1] works like the threading module, except that it runs callables in subprocesses instead of threads. To pass python objects to your process you can use the primitives offered by the modules (Queue, Pipe, Connection...). Here is a minimal example using Queue : Child process implementation (let's call it my_twisted_module.py): -- from twisted.internet import reactor, task dispatch_task = None in_queue = None out_queue = None def recv(): obj = in_queue.get_nowait() [do whatever you want with obj] def send(obj): out_queue.put_nowait(obj) def twisted_process(inq, outq): global dispatch_task, in_queue, out_queue in_queue = inq out_queue = outq dispatch_task = task.LoopingCall(recv) dispatch_task.start(0.1) reactor.run() -- On the parent process side: -- from my_twisted_module import twisted_process [something similar to the child process implementation for communications] # Note that out_queue in this side is passed as in_queue to the other side and vice versa in_queue = Queue() out_queue = Queue() p = multiprocessing.Process(target=twisted_process, args=(out_queue, in_queue)) p.start() -- You can pass any python object through the queues as long as it is picklabe. Passing SA objects is probably not a good idea though as this whole mess is about isolating Twisted from SA :) In my project I organized the communications with a dispatcher (louie [2]), and I pass signals through the queues. This way I have a well defined protocol to communicate with the sub process(es). Although I did not do any timing I think the communications between the processes must introduce some significant overhead, so it is probably not a good idea to do this if you pass objects all the time. In my case, the communications are limited to initialization calls, status keeping, etc... Luper [1]http://docs.python.org/library/multiprocessing.html [2]http://pylouie.org/ ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Plugin startup and shutdown actions
I would like to port my Application API based services to the twistd plugin interface, to be able to retrieve command line options. In the existing services, I reimplemented t.a.s.MultiService startService() and stopService() to do some actions at startup and shutdown. How can I do this in a plugin ? -- Lup ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Plugin startup and shutdown actions
2009/4/11 Jean-Paul Calderone : > On Sat, 11 Apr 2009 00:24:37 +0200, luper rouch > wrote: >> >> I would like to port my Application API based services to the twistd >> plugin interface, to be able to retrieve command line options. >> >> In the existing services, I reimplemented t.a.s.MultiService >> startService() and stopService() to do some actions at startup and >> shutdown. >> >> How can I do this in a plugin ? >> > > The top-level API for plugins is mostly a function that returns an IService > provider. So if you have your own version of MultiService, you can just > return an instance of that, just like the one you were creating in your .tac > file, from a makeService function which is registered as a plugin of the > suitable type. > Thanks I didn't think it was so simple ! I have a last question, I put my plugins in a 'twisted/plugins' subfolder of my project, and running them from the command line works fine. How can I invoke them in unit tests (I need to be able to start and stop them), since the plugin files are not in a package ? ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Plugin startup and shutdown actions
2009/4/11 Esteve Fernandez : > On Saturday 11 April 2009 19:42:31 Jean-Paul Calderone wrote: >> They can be imported from "twisted.plugins". For example, if you name your >> "dropin" file foo_plugins.py, then "from twisted.plugins import >> foo_plugins" should work and let you test any code that is part of your >> plugin definitions. Note that you should try to keep the amount of code and >> dependencies in a dropin file to a minimum, since this must all be loaded >> and executed whenever a search for any plugin is performed. > > Shouldn't twisted and twisted/plugins be non-importable? I thought twisted and > twisted/plugins must not have __init__.py files in order for the plugin > system to be able to find them (through getPlugins). Or did that behavior > change in recent versions? > > Cheers. It's working on 8.2.0 release, I have been able to port all my tests to the plugins API (they even seem less hackish now...) Thanks ! ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
