Thanks for the suggestions.
I've been looking at the source code of threading support objects and
I saw that non-blocking requests in queues use events, while blocking
requests just use InterlockedExchange.
So plain old put/get is much faster and I've managed to confirm this
today with further testing.
Sorry about the semicolon, just can't seem to shake it with my pascal
& C++ background :)
Currently, I've managed to get the code to this stage:
class mt(threading.Thread):
q = Queue.Queue()
def run(self):
dbf1 = Dbf('D:\\python\\testdbf\\promet.dbf', readOnly=1)
for i1 in xrange(len(dbf1)):
self.q.put(dbf1[i1])
dbf1.close()
del dbf1
self.q.put(None)
t = mt()
t.start()
time.sleep(22)
rec = 1
while rec <> None:
rec = t.q.get()
del t
Note the time.sleep(22). It takes about 22 seconds to read the DBF
with the 200K records (71MB). It's entirely in cache, yes.
So, If I put this sleep in there, the whole procedure finishes in 22
seconds with 100% CPU (core) usage. Almost as fast as the single
threaded procedure. There is very little overhead.
When I remove the sleep, the procedure finishes in 30 seconds with
~80% CPU (core) usage.
So the threading overhead only happens when I actually cause thread
interaction.
This never happened to me before. Usually (C, Pascal) there was some
threading overhead, but I could always measure it in tenths of a
percent. In this case it's 50% and I'm pretty sure InterlockedExchange
is the fastest thing there can be.
My example currently really is a dummy one. It doesn't do much, only
the reading thread is implemented, but that will change with time.
Reading the data source is one task, I will proceed with calculations
and with a rendering engine, both of which will be pretty CPU
intensive as well.
I'd like to at least make the reading part behave like I want it to
before I proceed. It's clear to me I don't understand Python's
threading concepts yet.
I'd still appreciate further advice on what to do to make this sample
work with less overhead.
--
http://mail.python.org/mailman/listinfo/python-list
