Hey all,
I've been working on an Java-style futures implementation in Python.
Futures are a way of representing asynchronous operations e.g.
operations that are run in another thread or process. The are are a easy
but powerful way of parallelizing sequential operations. The also
provide a consistent interface across implementations e.g. they can
provide the same interface to threading and to multiprocessing.
For example:
def is_prime(n):
"Return True iff n is prime"
...
def check_primes(numbers):
return map(is_prime, numbers)
Could be parallelized as:
def check_primes(numbers):
# ProcessPoolExecutor will create one worker process
# per CPU if called without arguments. Using threads
# is valueless because of the GIL.
with futures.ProcessPoolExecutor() as executor:
return executor.map(is_prime, numbers)
A more complex example:
def load_url(url, timeout):
return urllib.request.urlopen(url, timeout=timeout).read()
### Download the content of some URLs - ignore failures.
def download_urls(urls, timeout=60):
url_to_content = {}
for url in urls:
try:
url_to_content[url] = load_url(url, timeout=timeout)
except:
pass
return url_to_content
Could be parallelized as:
# Making this a global variable used any many functions ensures that
# the global thread count is kept under control.
executor = futures.ThreadPoolExecutor(50)
def download_urls(urls, timeout=60):
url_to_content = {}
# Run load_url for every url and get the result as a FuturesList.
fs = executor.run_to_futures(
(functools.partial(load_url, url, timeout) for url in urls),
timeout=timeout)
for url, future in zip(urls, fs.successful_futures()):
url_to_content[url] = future.result()
return url_to_content
The Python 3.0+ code is here:
http://code.google.com/p/pythonfutures/source
Any feedback on the API would be very much appreciated!
Cheers,
Brian
--
http://mail.python.org/mailman/listinfo/python-list
