On Thursday, October 10, 2013 11:01:25 PM UTC-7, Peter Cacioppi wrote:
> Could someone give me a brief thumbnail sketch of the difference between
> multi-threaded programming in Java.
>
>
>
> I have a fairly sophisticated algorithm that I developed as both a single
> threaded and multi-threaded Java application. The multi-threading port was
> fairly simple, partly because Java has a rich library of thread safe data
> structures (Atomic Integer, Blocking Queue, Priority Blocking Queue, etc).
>
>
>
> There is quite a significant performance improvement when multithreading here.
>
>
>
> I'd like to port the project to Python, partly because Python is a better
> language (IMHO) and partly because Python plays well with Amazon Web
> Services.
>
>
>
> But I'm a little leery that things like the Global Interpret Lock will block
> the multithreading efficiency, or that a relative lack of concurrent off the
> shelf data structures will make things much harder.
>
>
>
> Any advice much appreciated. Thanks.
I should add that the computational heavy lifting is done in a third party
library. So a worker thread looks roughly like this (there is a subtle race
condition I'm glossing over).
while len(jobs) :
job = jobs.pop()
model = Model(job) # Model is py interface for a lib written in C
newJobs = model.solve() # This will take a long time
for each newJob in newJobs :
jobs.add(newJob)
Here jobs is a thread safe object that is shared across each worker thread. It
holds a priority queue of jobs that can be solved in parallel.
Model is a py class that provides the API to a 3rd party library written in C.I
know model.solve() will be the bottleneck operation for all but trivial
problems.
So, my hope is that the GIL restrictions won't be problematic here. That is to
say, I don't need **Python** code to ever run concurrently. I just need Python
to allow a different Python worker thread to execute when all the other worker
threads are blocking on the model.solve() task. Once the algorithm is in full
swing, it is typical for all the worker threads should be blocking on
model.Solve() at the same time.
It's a nice algorithm for high level languages. Java worked well here, I'm
hoping py can be nearly as fast with a much more elegant and readable code.
--
https://mail.python.org/mailman/listinfo/python-list
