Re: [Twisted-Python] Application Design help - Concurrent but not Protocols based.

2009-06-10 Thread Senthil Kumaran
On Wed, Jun 03, 2009 at 02:55:47PM -0400, John Santos wrote: > > This is not an issue specifically related to Python or Twisted, but > there is a very serious synchronization issue that needs to be > addressed with this application design. (Trust me, I've seen this > issue come up dozens of times

Re: [Twisted-Python] Application Design help - Concurrent but not Protocols based.

2009-06-10 Thread Senthil Kumaran
On Wed, Jun 03, 2009 at 10:43:34AM -0700, Minesh Patel wrote: > For this, since your data is huge and seems like it will need a lot of > CPU utilization, you might have to deferToThread, since Twisted runs > in a single thread and will block AFAIK. > > d = threads.deferToThread(do_a, arg1_to_do_a

Re: [Twisted-Python] Application Design help - Concurrent but not Protocols based.

2009-06-03 Thread glyph
On 3 Jun, 04:25 pm, orsent...@gmail.com wrote: >My Question: Can this be designed in way that looking for new files is >also asynchronous activity? Sure. >What will be the deferred in this case? >Now, after reading the contents, I will have to do a non-blocking call >to fetch data, either using

Re: [Twisted-Python] Application Design help - Concurrent but not Protocols based.

2009-06-03 Thread David Ripton
On 2009.06.03 21:55:27 +0530, Senthil Kumaran wrote: > 1) I need to constantly monitor a particular directory for new files. > 2) Whenever a new file is dropped; I read that file and get > information on where to collect data from that is a) another machine b) > machine2-different method c) databas

Re: [Twisted-Python] Application Design help - Concurrent but not Protocols based.

2009-06-03 Thread John Santos
On Wed, 3 Jun 2009, Senthil Kumaran wrote: > Hello Twisted Developers/Users, > > This is my first concurrent application design and my first trial with > twisted. I have read the documentation and understand where twisted > plays its part. Unfortunately, I could not directly relate it to my > req

Re: [Twisted-Python] Application Design help - Concurrent but not Protocols based.

2009-06-03 Thread Minesh Patel
> My Application Details: > > 1) I need to constantly monitor a particular directory for new files. > 2) Whenever a new file is dropped; I read that file and get > information on where to collect data from that is a) another machine b) > machine2-different method c) database. > 3) I collect data fr

[Twisted-Python] Application Design help - Concurrent but not Protocols based.

2009-06-03 Thread Senthil Kumaran
Hello Twisted Developers/Users, This is my first concurrent application design and my first trial with twisted. I have read the documentation and understand where twisted plays its part. Unfortunately, I could not directly relate it to my requirements and hence, could not go forward with designing