I'm reading and processing a huge file, so during the execution I want to
now the state of the processing: how many lines are already processed, and
so on. The first approach is:
f = open(filename)
n = 0
for l in f:
if n % 1000 = 0:
print "Reading %d lines" %n
do_something(l)
but I want something diffent. I want to monitor the running of the computing
every n seconds. It's the first time I use threading, and I think mine is
not the best solution:
import threading
import time
import Queue
class Reader():
def __init__(self,filename):
self.filename = filename
self.lineno = 0
def __iter__(self):
f = open(self.filename)
for line in f:
self.lineno += 1
time.sleep(0.01) # slow down
yield line
f.close()
class Monitor(threading.Thread):
def __init__(self,reader,stop_queue,interval=2):
threading.Thread.__init__(self)
self.interval = interval
self.reader = reader
self.stop_queue = stop_queue
def run(self):
while True:
try:
if self.stop_queue.get(timeout=self.interval) == "stop":
break
except Queue.Empty:
pass
print "MONITOR: ", reader.lineno
reader = Reader("r1_200910.log")
q = Queue.Queue()
monitor = Monitor(reader,q)
monitor.start()
for line in reader:
pass
# do_somethinghard(line)
q.put("stop")
monitor.join()
It't works, but I don't like how I'm stopping the thread. In general the
Monitor class can be more complex, for example a progress bar. Using python
2.6
--
http://mail.python.org/mailman/listinfo/python-list
