Hello the list, I have a, let say 'special' , request. I would like to make a very specific type of distributed computing. Let me explain myself :
I have a HUUUUUGE(everything is relative) amount of data coming from UDP/514 ( good old syslog), and one process alone could not handle them ( it's full of regex, comparison and counting. all this at about 100K lines of data per seconds and each line is about 1500o long. I tried to handle it on one process, 100% core time in htop. ) So i said to myself : why not distribute the computing ? I looked for a good library that could do the distribution/process management job for me (http://wiki.python.org/moin/ParallelProcessing). What i found was not really usable by me ( distributed processing with a lot of processing but almost no process communication wich is the opposite of the thing i would like to do: distribute the events in real time so it is processed in real time.). My question is : is their any library or API or anything that is able to do that ? Sincerly Julien PS: This is my first time posting on any mailing list, my explanation/syntax could look a little bite, messy.
-- http://mail.python.org/mailman/listinfo/python-list
