Re: [Twisted-Python] Application Design help - Concurrent but not Protocols based.
On Wed, Jun 03, 2009 at 10:43:34AM -0700, Minesh Patel wrote: > For this, since your data is huge and seems like it will need a lot of > CPU utilization, you might have to deferToThread, since Twisted runs > in a single thread and will block AFAIK. > > d = threads.deferToThread(do_a, arg1_to_do_a, arg2...) > d.addCallback(do_a2, ...) > I delayed and forgot to respond to this thread. Sorry for that. Actually, I had asked the question on doing asynchronous monitoring of new file arrival and then based on the file contents, doing tasks a,b,c asynchronously. Thank you for your responses. I got a number of pointers to look at. * As a newbie to this world, I just started studying further. I realized that in my application, I have the control form the client on when I can notify that a particular file can be looked at. So, for the time being, I just skipped the inotify linux system related calls ( BTW, Twsited + inotify works great) and when ahead with just implementing a Twisted a Server which listens to a particular port and when receives data, will either read the file or parse the data to do some activity (like fetch logs from database). threads.deferToThread(func,[args]) is what I am using. http://paste.pocoo.org/show/122155/ I am still working out more details as doing this threads.deferToThread calls effectively, because there could be number of clients parallely requesting it and the data from my query (internal function) could be so big that it might take 30 minutes to service a request. * Another thing is when using something like "from datasources import query" I could not make it to application (.tac to run it from twistd), it could nto find datasources in its PYTHONPATH. Got to figure out how to do it, which I might pretty soon. Thanks again, Senthil ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Application Design help - Concurrent but not Protocols based.
On Wed, Jun 03, 2009 at 02:55:47PM -0400, John Santos wrote: > > This is not an issue specifically related to Python or Twisted, but > there is a very serious synchronization issue that needs to be > addressed with this application design. (Trust me, I've seen this > issue come up dozens of times in over 30 years of experience...) All the points which you have mentioned are very true. I also see that the kind of problems we might land up in if not are using certain filesystem provided facilties. If I were go for the file presence notification, I might use inotify, which seems to be pretty good. Thanks, Senthil ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] stop conditions and deferToThread()
Hi As discussed in a previous thread http://twistedmatrix.com/pipermail/twisted-python/2009-May/019717.html a task put in its own thread via deferToThread() won't stop even though the reactor has stopped. It has to carry out its own check that the reactor is running and stop if that is not the case. My question is: Is it possible to use anything else than the reactor.running as stop condition in this way? In my current twisted application I would like to keep the reactor running but still have some deferToThread jobs stop if e.g. an exception occurs in one of the jobs. In the code example below I try to use my own global field, running, as stop condition, but it doesn't work. If I let bar() call reactor.stop() after the time.sleep(3) and use reactor.running in foo() it works, however. import time from twisted.internet import reactor from twisted.internet.threads import deferToThread running = True def foo(): while running: print "working" time.sleep(1) def bar(): time.sleep(3) print "stopping" running = False d1 = deferToThread(foo) d2 = deferToThread(bar) reactor.run() Best regards, Thomas Jakobsen ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] stop conditions and deferToThread()
Thomas Jakobsen wrote: > Hi > > As discussed in a previous thread > >http://twistedmatrix.com/pipermail/twisted-python/2009-May/019717.html > > a task put in its own thread via deferToThread() won't stop even > though the reactor has stopped. It has to carry out its own check that > the reactor is running and stop if that is not the case. > > My question is: Is it possible to use anything else than the > reactor.running as stop condition in this way? In my current twisted > application I would like to keep the reactor running but still have > some deferToThread jobs stop if e.g. an exception occurs in one of the > jobs. A threading.Event object? Can you run the jobs as sub-processes? Then you could just kill them. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] stop conditions and deferToThread()
I'm not sure this is the problem you're facing, but... On Wed, Jun 10, 2009 at 12:04:16PM +0200, Thomas Jakobsen wrote: > import time > from twisted.internet import reactor > from twisted.internet.threads import deferToThread > > running = True > > def foo(): >while running: >print "working" >time.sleep(1) > > def bar(): global running >time.sleep(3) >print "stopping" >running = False > > d1 = deferToThread(foo) > d2 = deferToThread(bar) > > reactor.run() ...does that make things run better? ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] SerialPort.loseConnection() BUG
On Wed, 10 Jun 2009 11:20:15 +0800, biziap biziap wrote:
Please don't top-post.
>Maybe you can try to call transport.flushInput(), transport.flushOutput()
>before loseConnection().
>
No, this is wrong. Do not call these methods, ever.
>
>2009/6/9 Nestor A. Diaz :
>> Hello, there is a bug on the SerialPort Win32 implementation (don't know
>> about the others), when i call the loseConnection method, it returns:
>>
>> twisted matrix exceptions.AttributeError: 'SerialPort' object has no
>> attribute '_tempDataBuffer'
>>
>> if i wrap the serialport class and put a :
>>
>> self._tempDataBuffer = ''
>>
>> in the __init__ method, i get:
>>
>> raise NotImplementedError("%s does not implement writeSomeData" %
>> exceptions.NotImplementedError: __main__.MySerialPort does not implement
>> writeSomeData
>>
>> however the port gets closed the right way.
>>
Can you provide a minimal example which demonstrates this behavior? It is
perhaps a bug in Twisted, but it is difficult to know without being able to
see *exactly* how the behavior is triggered.
Thanks,
Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] stop conditions and deferToThread()
On Wed, 10 Jun 2009 12:04:16 +0200, Thomas Jakobsen wrote: >Hi > >As discussed in a previous thread > > http://twistedmatrix.com/pipermail/twisted-python/2009-May/019717.html > >a task put in its own thread via deferToThread() won't stop even >though the reactor has stopped. It has to carry out its own check that >the reactor is running and stop if that is not the case. deferToThread is optimized for short-running tasks. Thread creation costs are amortized over the lifetime of the process. This is because the intended use of deferToThread is to run functions which compute some result and then return, /not/ for functions which run forever (or at least until the process is ready to exit). You should try to find a way to re-arrange your code so that there is no need to check if the reactor is still running - just do one piece of work in a thread at a time. When your non-threaded application code gets the result back, if no one has asked it to shut down, then it can ask for another job to be executed. >My question is: Is it possible to use anything else than the >reactor.running as stop condition in this way? Also, I want to point out that while `reactor.running´ isn't private (since it doesn't start with an underscore, as Twisted's privacy policy dictates it would need to), it still isn't something you really want to be using. It's not part of any interface, which means there is no guarantee it will be provided by all reactors. It's also not really for you. It's a flag that the reactor uses to track its own internal state. Its meaning may not actually correspond to the meaning you'd like to assign to it. Whenever you're trying to do something like this (which I still don't recommend), you should create your own state tracking and use that instead. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Need working examples of imap4 client.
Can someone point me to a working example of imap4 client? I've tried the example off the site, code from Abe Fettig's book, and every example I could find on the web. (most of them seem outdated, as do most discussions I've found on the subject) All of them invariably end with: twisted.internet.error.TimeoutError: User timeout caused connection failure or twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion. I've pointed the code at two different IMAP servers, both of which (one is gmail, the other is a private server) I am able to access with a hitch with python's own imaplib. I've tried the sample code on an Ubuntu system, OSX, and tried python versions 2.5 and 2.6. Ideally, I'd love to see a snipped which is able to log into an imap server and gets a list of mailboxes. If the example on the site works for other folks, I'd love a nudge in the right direction to modify the code to get a bit more debugging info. Thanks in advance! ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] SerialPort.loseConnection() BUG
On 10 Jun, 12:23 pm, exar...@divmod.com wrote: >Please don't top-post. Since I'm seeing more and more messages where someone is top-posting, I think it might be helpful if we sent a more specific example of a positive example of what replies are expected to look like. I think for now it might be good to say we prefer this style: http://en.wikipedia.org/wiki/Posting_style#Inline_replying anybody else have a better reference? ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
