Hi, I am in the process of evaluating a tool for the newtorking architecture I need. And wanted to ask the group questions, I could not easily get answered from reading docs and various posts:
Basically my architecutre requires the following a) multi-CPU/multi-core scalability b) multi-machine horizontal scalability c) Event dispatching: having a single 'dispatcher' instance being able to read rows (events) from postgresql database and being able to dispatch it to available 'worker' processes d) chat server For c) and d) am looking for a framework such that it will allow me to develop the 'worker' processes such that they reside on mulitple machines, and a lot of the 'hard work' of registration, message passing, restarting, SNMP compatibilty . Twisted appears to already have message passing, chat server and many many other neat things that if I do not need now -- probably will need in the future. However, I cannot quite understand how the 'multi-process' part is supported. I have read this: http://www.python.org/workshops/2002-02/papers/09/index.htm (found it via Bruce Eckel's log) and it says that: "... and since forking Python processes has many disadvantages, like Python's reference counting not playing well with copy-on-write and problems with shared state, it was felt the best option was an event-driven framework. " I also looked at the docs for the http://twistedmatrix.com/projects/core/documentation/howto/process.html But I got confused there Basically I am looking to 'pre-spawn' a number of worker threads configured from command line (and each of the worker threads will precreate a database connection). So each multi-cpu machine will have one twisted Server, each server will pre-spawn N worker processes and each process will have its own database connection (and therefore each can do its own caching and transaction control (or use memchaced server) to share cache). I did not see however how to: a) make the Deferred mechanism to pass the event data received in the Asynch loop to one of the worker processes (there are appeared to be no 'inheritance' structure to where derive the worker processes from). b) I did not see how multiple twisted servers, each running on a separate multi-cpu machine can register together to be in one 'cluster' -- so that my dispatcher process that reads the events from the database can 'round-robin' the events to them. c) I did not see if twisted 'figures out' that given process runs on a local machine vs remote and optimises the IPC communication for local IPC. Because I did not see the above -- it lead me to believe that I am trying to ask Twisted have something it was not meant to do (My view of the architecture is somewhat similar to how would ACE/TAO ORB notification service would work -- as I am familiar with those). I kind of did not go into the multi-box architecutre of how the Twisted chart server is working, because I thought I need to understand the above first. So wanted to ask you guys if I am looking for the right documentation or may be there is a separate sub-project that is doing what I am looking for. thanks in advance, Vlad -- V S P torea...@fastmail.fm -- http://www.fastmail.fm - Or how I learned to stop worrying and love email again _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
