I have been experimenting with some thread programming, but as I'm
doing this on my own I am worried I might be making a major mistake.
Here's a brief rundown of what I am working on. Multiple threads, via
Queue, are used to perform RBL checks on an IP. The threads are passed
a defined class (ConnectionScore) in their init. This is used to
collect the results, including a method to add the result. If I run
the program, everything goes fine and once all the threads complete the
main program has all the data from their RBL checks in the
ConnectionScore class.
It seems to work, even with an obscene (and silly) number of WORKERS
and RBLs to check. But I was reading in the Python Cookbook this
evening and it raised the issue in my mind of what happens if multiple
threads try to add data to the class at the same time.
I've attached the code below. The ConnectionClass.addRBL method just
takes the result and adds it to a dictionary object attached to the
class.
#!/usr/bin/python
from UCE_weight import ConnectionScore
from rbl import *
import Queue
import threading
import time, random
starttime = time.time()
WORKERS=5
class Worker(threading.Thread):
def __init__(self,queue,score):
self.__queue = queue
threading.Thread.__init__(self)
self.score = score
def run(self):
while 1:
RBL = self.__queue.get()
# Are we at the end of the queue?
if RBL is None: break
self.result = checkRBL("127.0.0.2",RBL,True,5)
score.addRBL(RBL,self.result[0],self.result[1])
queue = Queue.Queue(0)
score = ConnectionScore()
f = open("rbl.list",'r')
MEGARBL = []
while 1:
line = f.readline().strip()
if not line: break
MEGARBL.append(line)
f.close()
for i in range(WORKERS):
Worker(queue,score).start() # start a worker
for i in MEGARBL:
queue.put(i)
for i in range(WORKERS):
queue.put(None) # add end of queue marker
while 1:
# wait for the queue???
if queue.empty():
break
else:
time.sleep(0.1)
elapsed = time.time() - starttime
rate = len(MEGARBL)/elapsed
rate = str(round(rate,2))
elapsed = str(round(elapsed,2))
score.showRBL()
print "Checked",str(len(MEGARBL)),"zones in",elapsed,"seconds.
(",rate.strip(),"per second )"
--
http://mail.python.org/mailman/listinfo/python-list
