----- Original Message -----
From: "Kent Johnson" <[EMAIL PROTECTED]>
To: "Tim Kelsey" <[EMAIL PROTECTED]>
Cc: <tutor@python.org>
Sent: Tuesday, October 28, 2008 12:45 PM
Subject: Re: [Tutor] Multi-threading IO and the GIL
On Tue, Oct 28, 2008 at 7:27 AM, Tim Kelsey <[EMAIL PROTECTED]> wrote:
Im not sure if this is the correct list to post this sort of thing
We'll try to answer pretty much any Python question.
Thanks very much for your response, im very confused by this and greatfull
for the input.
but I
have a question about python threading and the GIL. I have a
multi-threaded
application that is hitting a MySQL database for some info from within
the
various threads. When increase the size of my thread pool I see little or
no
improvement in the application performance but the CPU usage is not
maxed.
This suggests to me that the app is IO bound on the database, surly more
threads should help in this case.
I can think of a few other possibilities...
Does the app actually create more threads? If you only create two
threads, the size of the thread pool doesn't much matter.
The app was tested with 5 to 20 threads in the pool, this is running on a
Linux box and 'top -H' shows the threads using CPU time. The odd behaviour
is that adding more threads has little impact on performance, suggesting
that the app is processor bound. However, the total CPU time is less than
100% ( ~60 - 80% ) seemingly indicating that IO is the bottleneck.
It's possible that all the threads are blocking on the database, for
example if they are all trying to modify the same row. Or the app
could truly be I/O bound, if it is reading large amounts of data from
the DB then more threads trying to read more data from the same source
may not help.
The SQL in this case is simply a very SELECT opperations returning only a
few K of data each time. The database is not under stress ( aprox 30-50% CPU
use )
Is it possible that the GIL is not being release by a thread during the
time
its blocking on the DB connection? Im using MySQLdb module and the
application is purely written in python.
I don't know but it seems unlikely. Is there a MySQLdb list where you
could ask?
Ill ask about and see what I can find
Kent
This email has been scanned for all known viruses by the Email Protection
Agency.
Many thanks,
Tim Kelsey
This e-mail is confidential and may be privileged. It may be read, copied and used only by the intended recipient. No communication sent by e-mail to or from Eutechnyx is intended to give rise to contractual or other legal liability, apart from liability which cannot be excluded under English law.
This email has been scanned for all known viruses by the Email Protection
Agency.
www.eutechnyx.com Eutechnyx Limited. Registered in England No: 2172322
_______________________________________________
Tutor maillist - Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor
