Re: [Twisted-Python] Twisted developers: please donate your time!
I will spend about two weeks to update code and english docs for the http://fats.burus.org projectand about 4 weeks to contribute abstract database synchronizer engine (dbsync) based on the twisted for the twisted-python community under the MIT license. I have negotiations with our organization(government media company in the Russia) to make it possible. On Tue, Mar 3, 2009 at 10:05 PM, Paul Swartz wrote: > On Tue, Mar 3, 2009 at 12:41 AM, wrote: > > > > On 2 Mar, 02:08 pm, ita...@itamarst.org wrote: > >> > >> On Tue, 2009-02-24 at 21:25 +1300, Michael Hudson wrote: > >>> > >>> 2009/2/24 Itamar Shtull-Trauring : > >>> > As part of the TSF's fundraising efforts, we are trying to get > upfront > >>> > donations of time you will spend developing Twisted. It will then get > >>> > matched by a donor, if we are successful in getting this grant. So if > >>> > you're planning on working on Twisted anyway this year, your work > will > >>> > count twice as much! > >>> > > >>> > So, if you're interested, please reply, saying something like "I will > >>> > spend two weeks working on Gopher support over the next year." > >>> > >>> I don't entirely understand, but I will commit to spending the > >>> equivalent of two working weeks as a general review monkey over the > >>> next year. Is that the sort of thing you wanted to hear? > >> > >> Yes - thanks to you and all thee rest of the people who are > >> volunteering! Anyone else willing to step up? > > I will commit to spending 2 weeks on Conch in the coming year. > > -p > -- > Paul Swartz > paulswartz at gmail dot com > http://paulswartz.net/ > AIM: z3penguin > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Using sqlalchemy in twisted.
Hi, all I am using sqlalchemy in twisted in my project in the way below. Defer any database operation so the twisted's main thread won't be blocked. And I use scoped_session, so that sessions won't have to be created again and again. == class Database() def __init__(self, conn_str): self.conn_str = conn_str self.engine = create_engine(self.conn_str, echo=False) self.Session = scoped_session(sessionmaker(bind = self.engine, expire_on_commit=False)) def getObjectById(self, klass, id): return threads.deferToThread(self._getObjectById, klass, id) def _getObjectById(self, klass, id): sess = self.Session() return sess.query(klass).get(id) == The code doesn't work. When I limit the thread numbers to 1 reactor.suggestThreadPoolSize(1) Everything goes fine. Other wise the server would be blocked and must be killed by "kill 9 ...". The result conflicts with my understanding of sqlalchemy. Since I don't share any object between threads, there should be no problem! Ah It always have risk to use something you haven't tried before I think I have no choice but always set thread pool size to 1 ... -- look to the things around you,the immediate world around you, if you are alive,it will mean something to you ——Paul Strand ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted developers: please donate your time!
Itamar Shtull-Trauring wrote: Yes - thanks to you and all thee rest of the people who are volunteering! Anyone else willing to step up? I'll commit to one week to create and document another example for the web site. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] how to pass on the connection failed or connection lost error
Hi
let me first explain the application that I am developing. I have an
application that will use the twisted part as a plugin. this twisted part will
act as server and as client both.
my application call the plugin method to send data to server module using
connectTCP. now the problem is I can not pass on the connection failed
exception to my calling application.
To explain it further, here is the dummy code
class MyReactor(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
reactor.run(installSignalHandlers=0)
class MyFactory(ClientFactory):
def __init__(self):
self.msg = None
def setMsg(self, msg):
self.msg = msg
def connectionFailed(self, connector, reason):
#here i need to send the exception back to the user
class Plugin:
def __init__(self):
self.running = False
def send(self, msg):
reactor.callFromThread(self.clientsend, msg)
def clientsend(self, msg):
myfactory = MyFactory()
myfactory.setMsg(msg)
reactor.connectTCP ('localhost',)
if( not self.running):
mr = MyReactor()
mr.start()
self.running = True
def startServer(self):
reactor.listenTCP(,ServerFactory())
if( not self.running):
mr = MyReactor()
mr.start()
self.running = True
class PluginTest:
def __init__(self):
self.plugin = Plugin()
def start(self):
self.plugin.startServer()
def send(self,msg):
try:
self.plugin.send(msg)
except:
#I want to print send error here
# error could be connection fail
ptest = PluginTest()
#I m not starting the server so that I could get the connectionfail error
ptest.send('hi')
Can you suggest me any work around to get the exception in the PluginUser.send
method
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] how to pass on the connection failed or connection lost error
On Wed, 4 Mar 2009 06:39:59 -0800 (PST), khawar hasham wrote: Hi let me first explain the application that I am developing. I have an application that will use the twisted part as a plugin. this twisted part will act as server and as client both. my application call the plugin method to send data to server module using connectTCP. now the problem is I can not pass on the connection failed exception to my calling application. Since you're running the reactor in one thread and the rest of your application in another thread, your question is basically one of message passing. You've already found reactor.callFromThread which is good; I think you just want the slightly more featureful version, available in twisted.internet.threads, blockingCallFromThread. This is implemented in terms of callFromThread, but additionally allows the caller to get the result of the function being called - including waiting on a Deferred if the function being called returns one. If you switch to this, then you only need to make Plugin.clientsend return a Deferred which eventually fires with a result or a Failure. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Re: Twisted developers: please donate your time!
Hi Itamar and Colleages: >As part of the TSF's fundraising efforts, we are trying to get upfront >donations of time you will spend developing Twisted. It will then get >matched by a donor, if we are successful in getting this grant. So if >you're planning on working on Twisted anyway this year, your work will >count twice as much! I would happy to donate some time to developing Twisted. However since the only Twisted code I have really looked at is the reactor, I am not sure where I can contribute. Off-hand documentation is a safe bet. Perhaps you provide suggestions and I can figure out whether I can do it, when, and how much time I can devote? Cheers, Andrew ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Re: Twisted developers: please donate your time!
On Wed, 4 Mar 2009 09:34:54 -0800 (PST), Andrew Francis wrote: Hi Itamar and Colleages: As part of the TSF's fundraising efforts, we are trying to get upfront donations of time you will spend developing Twisted. It will then get matched by a donor, if we are successful in getting this grant. So if you're planning on working on Twisted anyway this year, your work will count twice as much! I would happy to donate some time to developing Twisted. However since the only Twisted code I have really looked at is the reactor, I am not sure where I can contribute. Off-hand documentation is a safe bet. Perhaps you provide suggestions and I can figure out whether I can do it, when, and how much time I can devote? http://twistedmatrix.com/trac/query?status=new&status=assigned&status=reopened&keywords=~documentation&order=priority shows all of the open tickets with the "documentation" keyword. So there's a lot to choose from, many are about the reactor. There's also one, #1009, about deferredGenerator/inlineCallbacks. I think you're somewhat familiar with those? It's also certainly the case that there are tickets open for reactor problems, so if you feel comfortable with that part of Twisted, there's plenty to contribute to. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Using sqlalchemy in twisted.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 4, 2009, at 3:28 AM, Peter Cai wrote: The code doesn't work. When I limit the thread numbers to 1 reactor.suggestThreadPoolSize(1) Everything goes fine. Other wise the server would be blocked and must be killed by "kill 9 ...". The result conflicts with my understanding of sqlalchemy. Since I don't share any object between threads, there should be no problem! I'm pretty sure you can't say this with full certainty and actually you are just wrong based on the code you showed. deferToThread doesn't use the same thread every time you call it, there's absolutely no guarantee in that and sqlalchemy keeps state around in that thread related to that object it returned. If you want to do any operations you need either to detach the object from the session before returning it and do any modification on the same object in another thread after reattaching it (pretty cumbersome). Or write your own threadpool that gives names to threads so that you can have a guarantee that you always call the same thread when working with a bunch of objects. Nonetheless though sqlalchemy is a huge project and I'm pretty sure it has some deadlocks and race conditions around which means that even taking care of these issues you'll have other problems when dealing with the orm. If you want to use sqlalchemy don't use its orm but just the query builder, it's the only sane way to integrate with twisted. Which anyway IMHO is the best way to use it anyway because it uses a lot less memory since it doesn't have to always cache objects because you control everything and can make that call when you really need it. Ah It always have risk to use something you haven't tried before I think I have no choice but always set thread pool size to 1 ... Not entirely true. - -- Valentino Volonghi aka Dialtone Now running MacOS X 10.5 Home Page: http://www.twisted.it http://www.adroll.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iEYEARECAAYFAkmu0NoACgkQ9Llz28widGXBawCg32svBsLqsIRLzvzOThgR4sA0 5UkAoIgNfyUDPl9c0nwSud0sem3aKjz5 =2XIX -END PGP SIGNATURE- ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Using sqlalchemy in twisted.
I think SQLAlchemy's ORM might work fine with Twisted. Check out http://foss.eepatents.com/sAsync/ . sAsync doesn't appear to be widely used, but I got the examples to run with some minor changes to the sqlite connection. I'm hoping to try something useful in the next week or two. On Mar 4, 2009, at 11:04 AM, Valentino Volonghi wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 4, 2009, at 3:28 AM, Peter Cai wrote: The code doesn't work. When I limit the thread numbers to 1 reactor.suggestThreadPoolSize(1) Everything goes fine. Other wise the server would be blocked and must be killed by "kill 9 ...". The result conflicts with my understanding of sqlalchemy. Since I don't share any object between threads, there should be no problem! I'm pretty sure you can't say this with full certainty and actually you are just wrong based on the code you showed. deferToThread doesn't use the same thread every time you call it, there's absolutely no guarantee in that and sqlalchemy keeps state around in that thread related to that object it returned. If you want to do any operations you need either to detach the object from the session before returning it and do any modification on the same object in another thread after reattaching it (pretty cumbersome). Or write your own threadpool that gives names to threads so that you can have a guarantee that you always call the same thread when working with a bunch of objects. Nonetheless though sqlalchemy is a huge project and I'm pretty sure it has some deadlocks and race conditions around which means that even taking care of these issues you'll have other problems when dealing with the orm. If you want to use sqlalchemy don't use its orm but just the query builder, it's the only sane way to integrate with twisted. Which anyway IMHO is the best way to use it anyway because it uses a lot less memory since it doesn't have to always cache objects because you control everything and can make that call when you really need it. Ah It always have risk to use something you haven't tried before I think I have no choice but always set thread pool size to 1 ... Not entirely true. - -- Valentino Volonghi aka Dialtone Now running MacOS X 10.5 Home Page: http://www.twisted.it http://www.adroll.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iEYEARECAAYFAkmu0NoACgkQ9Llz28widGXBawCg32svBsLqsIRLzvzOThgR4sA0 5UkAoIgNfyUDPl9c0nwSud0sem3aKjz5 =2XIX -END PGP SIGNATURE- ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Using sqlalchemy in twisted.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 4, 2009, at 11:15 AM, Chris Foster wrote: I think SQLAlchemy's ORM might work fine with Twisted. Check out http://foss.eepatents.com/sAsync/ . sAsync doesn't appear to be widely used, but I got the examples to run with some minor changes to the sqlite connection. I'm hoping to try something useful in the next week or two. sAsync repository hasn't changed since about 1 or 2 years ago. Now sqlalchemy 0.5 changed quite a bit of the internals and so on. Notice however that the tests for sAsync use sqlite that is limited essentially to 1 connection, which goes back to your original discontent. I tried to integrate sqlalchemy ORM in the past and it never worked right because the code in the objects didn't know that it was dealing with twisted and because sqlalchemy is mainly used in a single thread, it's thread safe but that doesn't mean that you can share objects between threads. - -- Valentino Volonghi aka Dialtone Now running MacOS X 10.5 Home Page: http://www.twisted.it http://www.adroll.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iEYEARECAAYFAkmu5KoACgkQ9Llz28widGWBWwCglXIKPUFaGbk5tI9XmtrH8lH+ 7woAnjrWWWEj2P78szZbNyLzSx0kgz4B =9uZo -END PGP SIGNATURE- ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted developers: please donate your time!
On Wed, Mar 4, 2009 at 11:20 PM, Florent Aide wrote: > Itamar Shtull-Trauring a écrit : > > So, if you're interested, please reply, saying something like "I will > spend two weeks working on Gopher support over the next year." > > > Hi all, > > I have worked in the last days on a demo application using twisted and > evolved from the simple pop3 + smtp exemple found in the Twisted book to add > a web interface (minimalistic but works) in nevow and SSL support + and > SQLAlchemy based SQL backend for passwords. > > I think it would be nice to write some complete tutorial based on this > example (and I'm ready to write it of course). But I'd like some > minimalistic review from a twisted master so that I don't tell too much > stupid things and don't spread bad practices. Sorry the mail went out before I could finish it... here is the URL of the project: http://bitbucket.org/faide/guineapy/overview/ Regards, Florent Aide. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted developers: please donate your time!
On Wed, Mar 4, 2009 at 2:22 PM, Florent Aide wrote: > On Wed, Mar 4, 2009 at 11:20 PM, Florent Aide wrote: >> Itamar Shtull-Trauring a écrit : >> >> So, if you're interested, please reply, saying something like "I will >> spend two weeks working on Gopher support over the next year." >> >> >> Hi all, >> >> I have worked in the last days on a demo application using twisted and >> evolved from the simple pop3 + smtp exemple found in the Twisted book to add >> a web interface (minimalistic but works) in nevow and SSL support + and >> SQLAlchemy based SQL backend for passwords. >> >> I think it would be nice to write some complete tutorial based on this >> example (and I'm ready to write it of course). But I'd like some >> minimalistic review from a twisted master so that I don't tell too much >> stupid things and don't spread bad practices. > > Sorry the mail went out before I could finish it... > > here is the URL of the project: > http://bitbucket.org/faide/guineapy/overview/ Very nice. Overall I think that the structure of the project looks good, especially the clean usage of Twisted's Application functionality. You mention "don't spread bad practices", and I think this brings up an important issue. It would be nice to have some of the main Twisted devs put a "best practices" stamp of approval on the code, and after putting on that stamp, clearly link to the code. This is something that would be good for the Twisted community and newcomers alike. -Alex > > Regards, > Florent Aide. > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > -- Alex Clemesha clemesha.org ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Using sqlalchemy in twisted.
I'm not quite sure, but I think I'm pretty careful of sharing objects between threads. 1st, I only cached as few as possible orm objects. I tried to detach them, but I found that if I detach them, I can't access any of their fields any more. 2nd, I create new orm objects based on client request, pass them to class Database and then merge them to scoped sessions, change, commit and then discard these objects. 3rd, I switch to sqlite frequently to check if there is any database operation outside Database, because sqlite doesn't allow multi-thread access. Actually it seems to work until 2 or 3 days ago suddenly cases hang the server. Ah, as I've already written lots of code in ORM, I think maybe I should try to change Database to use a dedicated thread to handle all database operations. That might be a bottle neck of my application, but I really can't give up orm as these mapper classes are used everywhere in my application. On Thu, Mar 5, 2009 at 3:04 AM, Valentino Volonghi wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > > On Mar 4, 2009, at 3:28 AM, Peter Cai wrote: > > The code doesn't work. When I limit the thread numbers to 1 >> >> reactor.suggestThreadPoolSize(1) >> >> Everything goes fine. Other wise the server would be blocked and must >> be killed by "kill 9 ...". >> >> The result conflicts with my understanding of sqlalchemy. Since I >> don't share any object between threads, there should be no problem! >> > > I'm pretty sure you can't say this with full certainty and actually you are > just wrong based on the code you showed. deferToThread doesn't use > the same thread every time you call it, there's absolutely no guarantee > in that and sqlalchemy keeps state around in that thread related to that > object it returned. If you want to do any operations you need either to > detach the object from the session before returning it and do any > modification > on the same object in another thread after reattaching it (pretty > cumbersome). > > Or write your own threadpool that gives names to threads so that you can > have a guarantee that you always call the same thread when working with > a bunch of objects. > > Nonetheless though sqlalchemy is a huge project and I'm pretty sure it has > some deadlocks and race conditions around which means that even taking > care of these issues you'll have other problems when dealing with the orm. > > If you want to use sqlalchemy don't use its orm but just the query builder, > it's the only sane way to integrate with twisted. Which anyway IMHO is the > best way to use it anyway because it uses a lot less memory since it > doesn't > have to always cache objects because you control everything and can make > that call when you really need it. > > Ah It always have risk to use something you haven't tried >> before >> I think I have no choice but always set thread pool size to 1 ... >> > > > Not entirely true. > > - -- > Valentino Volonghi aka Dialtone > Now running MacOS X 10.5 > Home Page: http://www.twisted.it > http://www.adroll.com > > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.9 (Darwin) > > iEYEARECAAYFAkmu0NoACgkQ9Llz28widGXBawCg32svBsLqsIRLzvzOThgR4sA0 > 5UkAoIgNfyUDPl9c0nwSud0sem3aKjz5 > =2XIX > -END PGP SIGNATURE- > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > -- look to the things around you,the immediate world around you, if you are alive,it will mean something to you ——Paul Strand ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Using sqlalchemy in twisted.
I changed my code to this style, it works again. This code would create a new session on each request and close it immediately. Hope this trick could save me! #= def require_session(f): '''create and close session for each synchronous method''' def wrapper(model, *args, **kw): sess = model.Session() try: return f(model, sess, *args, **kw) finally: sess.close() return wrapper class Database() def __init__(self, conn_str): self.conn_str = conn_str self.engine = create_engine(self.conn_str, echo=False) self.Session = sessionmaker(bind = self.engine, expire_on_commit=False) def getObjectById(self, klass, id): return threads.deferToThread(self._getObjectById, klass, id) @require_session def _getObjectById(self, sess, klass, id): return sess.query(klass).get(id) #= On Thu, Mar 5, 2009 at 4:29 AM, Valentino Volonghi wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > > On Mar 4, 2009, at 11:15 AM, Chris Foster wrote: > > I think SQLAlchemy's ORM might work fine with Twisted. Check out >> http://foss.eepatents.com/sAsync/ . sAsync doesn't appear to be widely >> used, but I got the examples to run with some minor changes to the sqlite >> connection. I'm hoping to try something useful in the next week or two. >> > > > sAsync repository hasn't changed since about 1 or 2 years ago. Now > sqlalchemy 0.5 > changed quite a bit of the internals and so on. Notice however that the > tests for > sAsync use sqlite that is limited essentially to 1 connection, which goes > back to > your original discontent. > > I tried to integrate sqlalchemy ORM in the past and it never worked right > because > the code in the objects didn't know that it was dealing with twisted and > because > sqlalchemy is mainly used in a single thread, it's thread safe but that > doesn't mean > that you can share objects between threads. > > - -- > Valentino Volonghi aka Dialtone > Now running MacOS X 10.5 > Home Page: http://www.twisted.it > http://www.adroll.com > > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.9 (Darwin) > > iEYEARECAAYFAkmu5KoACgkQ9Llz28widGWBWwCglXIKPUFaGbk5tI9XmtrH8lH+ > 7woAnjrWWWEj2P78szZbNyLzSx0kgz4B > =9uZo > > -END PGP SIGNATURE- > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > -- look to the things around you,the immediate world around you, if you are alive,it will mean something to you ——Paul Strand ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
