The nice thing about using Python's process support is that you can spawn native Processes that run in separate heaps directly from ONE Python Twisted app. Not many running side-by-side, which adds the complexity of now coordinating among them (however easy with additional protocols like spread).
Inter-Process communication is also supported in Python's new multiprocessing package. And again, it can all be orchestrated from a single service _instance_. In my code, I need to run "on the metal" for some tasks and not others. Agreed, the event-based reactor threading in Twisted is great. But not for all modes of computation. For those, I offload onto OS processes directly onto CPU cores. Twisted does not provide a way to leverage its API against Python's support for this feature. So I have to find a way to marry the two. What I ended up doing was using the multiprocess package to kick off hard Process objects in a Python Process pool executing Python functions. Those functions make calls into Twisted, but for it to work, they had to start their own reactors because a Process has its own, separate OS memory, etc. Running compute intensive tasks in processes with their own memory makes a lot of sense for some things that Python cannot do with virtual machine thread contexts. Darren On Thu, 2010-02-25 at 10:21 -0800, K. Richard Pixley wrote: > Single threaded, event loop based code like twisted rocks hard. > > Once upon a time, threads were like that too and the distinction > between threads and event loops was grey. But with the advent of > mandatory preemptive thread scheduling and the ability to run multiple > threads on separate shared memory processors, the difference between > programming with threads and programming with parallel heavy weight > processes that share memory became extremely grey, (aside from the > problems debugging threads which don't exist for heavy weight > processes). > > Threads routinely use shared memory and shared memory (generally) > requires a common kernel. OTOH, message passing can use a common > kernel but can also extend out to other machines on the network. If > you use twisted for highly efficient "single thread/multiple task" > heavy weight processes, and something like spread, you end up with the > best of all worlds. Highly efficient, symmetric, network based > parallelism, with fault tolerance thrown in for free. > > My point here is that there are other ways to go about exploiting > symmetric multiprocessor machines, even banks of them, that neither > require threads, nor the multiprocessing package. > > --rich > > Darren Govoni wrote: > > > Looks interesting. I'm going to check out that package. > > > > > > My original request was more along the lines of using Python's new > > support for native CPU core's and processes (the multiprocessing > > package is for this). Python's built-in thread support has global > > lock constraints that underperform in some situations. > > > > But I ran into a problem using multiprocessing module with Twisted > > that was pointed out on the Twisted trac with pickling class methods > > and apparently Python's CPU threading support attempts to do this in > > some situations (e.g. when I try to pass a class method to a native > > thread). > > > > On Wed, 2010-02-24 at 12:04 -0800, K. Richard Pixley wrote: > > > > > Glyph Lefkowitz wrote: > > > > > > > On Feb 21, 2010, at 8:00 PM, Alexandre Quessy wrote > > > > > > > > > > > > > Hello everyone, > > > > > I have done something similar to this, but I used the children IO > > > > > stream to control them. Maybe I should have done that using some > > > > > higher level protocol, such as AMP or PB. > > > > > > > > > > > > > > > > > > Using a higher-level protocol is generally better, if for no other > > > > reason that it gives you a framework within which to document your > > > > design decisions. It's much easier to say "An AMP command with a 'foo' > > > > String argument and a 'bar' Integer argument" than to say "The first > > > > two bytes of the message are the length of the first argument. The > > > > next n bytes are the first argument. The first argument shall be > > > > interpreted as... (etc, etc)" > > > > > > > > > > I'm working on an interface right now to the spread toolkit, > > > (http://spread.org), which implements virtual synchrony, > > > (http://en.wikipedia.org/wiki/Virtual_synchrony). > > > > > > For distributed, symmetric, fault tolerant parallelism in small to > > > medium scale with high reliability, this might be an option. > > > > > > --rich > > > > > > _______________________________________________ > > > Twisted-Python mailing list > > > Twisted-Python@twistedmatrix.com > > > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > > > > > > > > > > > ____________________________________________________________________ > > > > _______________________________________________ > > Twisted-Python mailing list > > Twisted-Python@twistedmatrix.com > > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
