Hello everybody! I have a question.
I have a Django app running on Heroku. I need to run about 100 worker threads
there to do uploads/downloads simultaneously. A Heroku Dyno has only 512MB of
memory, so I'm reluctant to run 100 worker threads. (I've had Dynos crash from
lack of memory when using 6 threads before.)
I heard that the asyncio module is mature and ready for usage, and I was happy
because I kept hearing about it in the last year, and I saw Guido's lecture
about it. If I understand correctly it would let me run multiple uploads and
downloads efficiently in one thread, which would conserve more resources than
using threads. (Please correct me if I'm wrong.)
Now, I am a little clueless about the whole way it's built, using coroutines
and tricky usage of `yield from`. I figured that since this is eventually a
library for concurrency, i.e. doing many tasks at the same time, there will be
an API in a style of "Here are 100 tasks for you to do concurrently, let me
know when they're done."
I looked at the asyncio documentation page and saw that it does mention futures
and executors, which is my favorite interface for doing concurrency. I was
happy and I skimmed the docs. But, I couldn't find a simple way to use these. I
don't want to learn how to define coroutines and use `yield from` to switch
between them. (I use `yield from` regularly and fully understand how it works,
I just don't write my programs that way.)
What I'm expecting is something like this:
download_file = lambda url: requests.get(url).content
urls = ['http://google.com/file1.jpg', 'http://google.com/file2.jpg',
'http://google.com/file3.jpg'] # etc.
with AsyncIOExecutor() as asyncio_executor:
files = asyncio_executor.map(download_file, urls)
And that's it, no coroutines, no `yield from`. Since, if I understand
correctly, asyncio requires a mainloop, it would make sense for the
AsyncIOExecutor to have a thread of its own in which it could run its mainloop.
Is this possible? Did someone implement this?
Thanks for your help,
Ram.
--
https://mail.python.org/mailman/listinfo/python-list
