Re: [Twisted-Python] running 1,000,000 tasks, 40 at-a-time

2011-10-26 Thread Jason Rennie
On Wed, Oct 26, 2011 at 11:08 AM, Jason Rennie wrote: > First off, have you read http://jcalderone.livejournal.com/24285.html ? >> If not, do. Take note of the line >> >> work = (callable(elem, *args, **named) for elem in iterable) >> >> work is a generator

Re: [Twisted-Python] running 1,000,000 tasks, 40 at-a-time

2011-10-26 Thread Jason Rennie
On Wed, Oct 26, 2011 at 10:22 AM, Terry Jones wrote: > Sounds like you should be using a Python generator and that you're not. > The issue I'm seeing is that the ~million DeferredSemaphore.run() calls are inefficient; it's independent of list/generator. First off, have you read http://jcalderon

[Twisted-Python] running 1,000,000 tasks, 40 at-a-time

2011-10-26 Thread Jason Rennie
The background: I've been using DeferredSemaphore and DeferredList to manage the running of tasks with a resource constraint (only so many tasks can run at the same time). This worked great until I tried to use it to manage millions of tasks. Simply setting them up to run (DeferredSemaphore.run(

Re: [Twisted-Python] Client/Server to stream text logfiles?

2011-09-29 Thread Jason Rennie
On Thu, Sep 29, 2011 at 10:37 AM, Victor Hooi wrote: > However, what did you mean by it may be "nontrivial to use twisted with > non-twisted friendly libraries"? What would make a library non-Twisted > friendly? > Most important, I think, is for non-reactor code to not block. E.g. if you're usi

Re: [Twisted-Python] Client/Server to stream text logfiles?

2011-09-27 Thread Jason Rennie
Sounds like your project would be appropriate for twisted. You'll have the easiest time if the reading and writing can each be implemented as a separate process (independent of other jobs on the machines). Be sure to read the following documentation: http://twistedmatrix.com/documents/current/co

Re: [Twisted-Python] How to safely stop a chain of deferreds

2011-06-29 Thread Jason Rennie
On Tue, Jun 28, 2011 at 9:30 AM, tomw wrote: > in a GUI application I'm using quite a comprehensive chain of deferreds > and I'm running the mainloop from the glib2reactor. At a certain point > of my application I need to terminate the entire chain of deferreds to > make sure that no more data is

Re: [Twisted-Python] problems with transport.write

2011-04-30 Thread Jason Rennie
the server side? If so, are you using callFromThread? http://pythonquirks.blogspot.com/2011/01/twisted-callwhenrunning-callfromthread.html Can you create a SSCCE (http://sscce.org/) and show it to us? Jason -- Jason Rennie Research Scientist, ITA Software 617-714-2645 ht

Re: [Twisted-Python] Understanding deferred and error handling

2011-04-20 Thread Jason Rennie
)". printTraceback does not return a meaningful value IIUC. I don't get any traceback either. > Your traceback went to stdout whereas you were probably watching stderr or a log file. Jason -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/

Re: [Twisted-Python] Twisted Plugins - Implementation Discussion

2011-04-07 Thread Jason Rennie
//mike.marineau.org/coil/ Jason -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/ ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [Twisted-web] Question regarding callLater() and service creation

2011-03-10 Thread Jason Rennie
On Thu, Mar 10, 2011 at 6:43 AM, Andrew Bennetts wrote: > There's nothing wrong or even particularly strange with using Twisted > for projects unrelated to networking. An event loop with good > facilities for spawning and interacting with subprocesses and calling > functions at certain times has

Re: [Twisted-Python] Error (and response) handling in protocols

2011-02-21 Thread Jason Rennie
ittle > package of things I think would make serial comms a happier experience > for all, and publish it somewhere. And maybe write about how to use > Twisted for non-internetty things. But I do need to actually > accomplish something first. In case you haven'

Re: [Twisted-Python] A thread in isolation

2011-02-17 Thread Jason Rennie
AgentServerFactory(protocol.ServerFactory): > > protocol = Producer > > > > > > class AgentClientFactory(protocol.ClientFactory): > > protocol = Consumer > > > > > > if __name__ == '__main__': > > for i in

Re: [Twisted-Python] What's the purpose of __?

2011-02-15 Thread Jason Rennie
__ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/ ___ Twisted-Python mailing l

Re: [Twisted-Python] sigchld

2011-02-15 Thread Jason Rennie
need to use the 'before' phase. During 'before', > Deferreds returned by event triggers are respected (i.e. the reactor keeps > running as long as they haven't fired). If you schedule one for 'during' or > 'after', the event trigger really needs to do all of its work and complete >

Re: [Twisted-Python] sigchld

2011-02-11 Thread Jason Rennie
e a better way to go. Thanks for the tip. Jason -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/ ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] sigchld

2011-02-11 Thread Jason Rennie
On Fri, Feb 11, 2011 at 10:01 AM, wrote: > On Python 2.5 and earlier you won't be able to get Twisted's SIGCHLD > handler using signal.getsignal. > We're also moving to python 2.6 (the change is really Debian 5 to Debian 6). Should it would work in 2.6? > What sort of things do your signal ha

[Twisted-Python] sigchld

2011-02-11 Thread Jason Rennie
r, is there an obvious flaw? In particular, can I get the SIGCHLD handler using signal.getsignal, then call it via my "super" handler? I don't need to do any special handling of SIGCHLD, so I could just leave that signal handler alone. All thoughts appreciated. Thanks, J

Re: [Twisted-Python] DeferredSemaphore - action on acquire and release

2011-02-08 Thread Jason Rennie
pointer > to DeferredLock, somehow I totally missed that. > > Cheers, > Jason > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-

Re: [Twisted-Python] DeferredSemaphore - action on acquire and release

2011-02-07 Thread Jason Rennie
d = self.sem.run(wrapper) >d.addBoth(self._release) >return d > > > I feel like there might be a simpler way to do this, but I just can't > see it. Or is this as simple as I can make it? > > — Jason > > _

Re: [Twisted-Python] design of twisted application

2011-02-02 Thread Jason Rennie
This is especially important if you start any processes from twisted (ProcessProtocol). Cheers, Jason -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/ ___ Twisted-Python mailing list Twisted-Python@twistedmat

Re: [Twisted-Python] Refactoring Documentation

2011-01-20 Thread Jason Rennie
#x27;s, especially the Tutorial/Library Reference/Language Reference breakdown. Users tend to know the level of understanding they are looking for and Python's documentation reflects that. Cheers, Jason -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.co

Re: [Twisted-Python] client connecting to 2 servers (nonsimultaneously)

2011-01-12 Thread Jason Rennie
at if the reactor is asleep waiting for > an event it will wake up soon, and then it will notice there are thread > call events to process. > > Jean-Paul > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http

Re: [Twisted-Python] client connecting to 2 servers (nonsimultaneously)

2011-01-12 Thread Jason Rennie
ead to jump to the front of the queue of threads to be run by python/linux? If there were no wakeUp in callFromThread, would the call simply be delayed until the thread round-robin got to the reactor thread? Or, without wakeUp, is it possible the callFromThread call would never be made? Cheers, Ja

Re: [Twisted-Python] client connecting to 2 servers (nonsimultaneously)

2011-01-12 Thread Jason Rennie
>... print 'foo running' >...>>> from twisted.internet import reactor >>>> reactor.callWhenRunning(foo) >foo running >>>> > Jean-Paul > > -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www

Re: [Twisted-Python] client connecting to 2 servers (nonsimultaneously)

2011-01-12 Thread Jason Rennie
Running in the __init__ method? > Why not calling directly reactor.connectTCP? > > Cheers, > > Benjamin > > -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/ ___ Twisted-Python mailing list Twisted-Pyt

Re: [Twisted-Python] client connecting to 2 servers (non simultaneously)

2011-01-11 Thread Jason Rennie
is a fantastic idea, though I would have used > rotate(-1), as your example looks like it rotate's "backwards" to me. > > Matter of taste I suppose. It doesn't make much actual difference. > > Kevin Horn > > > On Tue, Jan 11, 2011 at 7:11 AM, Jason Rennie

Re: [Twisted-Python] client connecting to 2 servers (non simultaneously)

2011-01-11 Thread Jason Rennie
d.internet import reactor > reactor.connectTCP(server1, 8010, factory) > reactor.run() > ******* > > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://tw

Re: [Twisted-Python] [newb] daemon question

2010-11-05 Thread Jason Rennie
ing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/ ___ Twisted-Python mailing list Twisted-Pytho

Re: [Twisted-Python] Twisted with wxPython in a thread

2010-08-24 Thread Jason Rennie
> Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/

Re: [Twisted-Python] handling SIGCHLD

2010-06-08 Thread Jason Rennie
ACfY8xqgU2LLOrs0cfR7HngUqmc > LOQAoKkur5+DK8lcZ+q6qWDtjqIpX3Ov > =XM/4 > -END PGP SIGNATURE- > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/

Re: [Twisted-Python] t.i.i.IProcessTransport

2010-06-01 Thread Jason Rennie
I created: http://twistedmatrix.com/trac/ticket/4479 Thanks for the pointers to the related bugs. I referenced them in #4479. Cheers, Jason On Tue, Jun 1, 2010 at 12:10 PM, Glyph Lefkowitz wrote: > > On Jun 1, 2010, at 11:53 AM, Jason Rennie wrote: > > > Is it a documen

[Twisted-Python] t.i.i.IProcessTransport

2010-06-01 Thread Jason Rennie
Cheers, Jason -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/ ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python