I have been experimenting with some thread programming, but as I'm doing this on my own I am worried I might be making a major mistake.
Here's a brief rundown of what I am working on. Multiple threads, via Queue, are used to perform RBL checks on an IP. The threads are passed a defined class (ConnectionScore) in their init. This is used to collect the results, including a method to add the result. If I run the program, everything goes fine and once all the threads complete the main program has all the data from their RBL checks in the ConnectionScore class. It seems to work, even with an obscene (and silly) number of WORKERS and RBLs to check. But I was reading in the Python Cookbook this evening and it raised the issue in my mind of what happens if multiple threads try to add data to the class at the same time. I've attached the code below. The ConnectionClass.addRBL method just takes the result and adds it to a dictionary object attached to the class. #!/usr/bin/python from UCE_weight import ConnectionScore from rbl import * import Queue import threading import time, random starttime = time.time() WORKERS=5 class Worker(threading.Thread): def __init__(self,queue,score): self.__queue = queue threading.Thread.__init__(self) self.score = score def run(self): while 1: RBL = self.__queue.get() # Are we at the end of the queue? if RBL is None: break self.result = checkRBL("127.0.0.2",RBL,True,5) score.addRBL(RBL,self.result[0],self.result[1]) queue = Queue.Queue(0) score = ConnectionScore() f = open("rbl.list",'r') MEGARBL = [] while 1: line = f.readline().strip() if not line: break MEGARBL.append(line) f.close() for i in range(WORKERS): Worker(queue,score).start() # start a worker for i in MEGARBL: queue.put(i) for i in range(WORKERS): queue.put(None) # add end of queue marker while 1: # wait for the queue??? if queue.empty(): break else: time.sleep(0.1) elapsed = time.time() - starttime rate = len(MEGARBL)/elapsed rate = str(round(rate,2)) elapsed = str(round(elapsed,2)) score.showRBL() print "Checked",str(len(MEGARBL)),"zones in",elapsed,"seconds. (",rate.strip(),"per second )" -- http://mail.python.org/mailman/listinfo/python-list