Oops, I forgot to start the thread in the above example for compute_something(), ``th.start()`` should go right after initializing the thread but before the try-finally block
On Thu, Feb 13, 2020 at 6:15 PM Kyle Stanley <[email protected]> wrote: > > I agree that is counter to the basic philosophy, though there are > potential use cases where it may be unavoidable to use threads. It seems > the best solutions when it is then are to use run_in_executor, which > automatically handles the waking up of the loop. > > While this doesn't provide an immediate solution, we're currently working > on implementing an asyncio.ThreadPool for Python 3.9, which is made to be > more self-contained (initialized and finalized within an async context > manager using ``async with``) and with a higher level API compared to > loop.run_in_executor(). In general, I'd recommend making use of either of > those two compared to directly using threads in asyncio. > > That being said, if you need more fine-grained control, there's also one > other alternative low-level construct/design pattern you can use for > working with threads in asyncio that we use in the implementation of some > coroutine methods, such as one that I recently added to 3.9 called > loop.shutdown_default_executor(). Using a similar format to your above > example: > > ``` > def _do_complete_something(fut, loop, ...): > try: > # perform some action, optionally get a result to set to the future > loop.call_soon_threadsafe(future.set_result, None) # replace None > with actual result if needed > except Exception as ex: > loop.call_soon_threadsafe(future.exception, ex) > > async def compute_something(...): > # In general, I'd recommend using get_running_loop() to get the event > loop within a coroutine function. > # It has more predictable behavior compared to get_event_loop(). > (Python 3.7+) > loop = asyncio.get_running_loop() > fut = loop.create_future() > th = threading.Thread(target=_do_compute_something, args=(fut,loop, > ...)) > try: > # use "return await future" if the value is desired > await future > finally: > th.join() > > async def compute_stuff(): > result = await compute_something(...) > > asyncio.run(compute_stuff()) > ``` > > Hopefully that helps a bit, or at least gives you some some ideas. > > On Thu, Feb 13, 2020 at 9:31 AM Brianvanderburg2 via Python-ideas < > [email protected]> wrote: > >> I agree that is counter to the basic philosophy, though there are >> potential use cases where it may be unavoidable to use threads. It seems >> the best solutions when it is then are to use run_in_executor, which >> automatically handles the waking up of the loop. While the Queue object >> doesn't work with threads, though there does seem to be a library janus >> which provides a thread-capbable queue object which could be another >> solution. >> >> >> -----Original Message----- >> From: Guido van Rossum <[email protected]> >> To: Brian Allen Vanderburg II <[email protected]> >> Cc: Python-Ideas <[email protected]> >> Sent: Thu, Feb 13, 2020 1:18 am >> Subject: Re: [Python-ideas] Asyncio Future.set_result_threadsafe >> >> Sharing futures between threads like that goes counter to asyncio's basic >> philosophy (which is not to use threads :-). >> >> You already showed the solution: >> future._loop.call_soon_threadsafe(future.set_result, ...). If that's >> unacceptable for you, maybe you can wrap the future in a wrapper class that >> calls call_soon_threadsafe. >> >> On Wed, Feb 12, 2020 at 8:51 PM Brian Allen Vanderburg II via >> Python-ideas <[email protected]> wrote: >> >> Currently asyncio.futures.Future.set_result will result in any callbacks >> being scheduled using loop.call_soon instead of >> loop.call_soon_threadsafe. However in situations where the future's >> result is set from a different thread, the loop might not wake up as a >> result of this if it is currently sleeping. >> >> >> def compute_something(...): >> loop = asyncio.get_event_loop() >> future = loop.create_future() >> manager.run_computation(future, ...) # add the computation to a >> thread that is running it >> return future >> >> async def compute_stuff(): >> result = await compute_something(...) >> >> loop = asyncio.get_event_loop() >> loop.run_until_complete(compute stuff()) >> loop.close() >> >> >> The reasoning behind it is after yielding the Future object, the ready >> list of the event loop is empty so when waiting for selectors the >> timeout value is None and it just waits. When the other thread calls >> set_result, the loop ready list will get updated, but it's still waiting >> on selectors. The other thread could call >> future._loop.call_soon_threadsafe(future.set_result, ...), which writes >> some bytes to a socket in order to wake the even loop. >> >> I'm aware there are other ways to do things like this (queues, >> executors, etc). It just seems that, regardless of any method used, the >> code holding the future with the purpose of setting a result on it >> should be able to be blind to other implementation details. >> future.set_result should be all it needs to know to call even if from >> another thread, or if not, a method like future.set_result_threadsafe, >> which would hide the details of the loop.call_soon/call_soon_threadsafe >> >> >> _______________________________________________ >> Python-ideas mailing list -- [email protected] >> To unsubscribe send an email to [email protected] >> https://mail.python.org/mailman3/lists/python-ideas.python.org/ >> Message archived at >> https://mail.python.org/archives/list/[email protected]/message/PLKYKFWM4EM7DHUHWOBDU2JSJM57X7GM/ >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> *Pronouns: he/him **(why is my pronoun here?)* >> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/> >> _______________________________________________ >> Python-ideas mailing list -- [email protected] >> To unsubscribe send an email to [email protected] >> https://mail.python.org/mailman3/lists/python-ideas.python.org/ >> Message archived at >> https://mail.python.org/archives/list/[email protected]/message/CWV6UQUN4LFZHQK53L463OY4D4EROFXS/ >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >
_______________________________________________ Python-ideas mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/QW5Q6Z5QCMIUBN5JEBDOCMTG2NRUG43A/ Code of Conduct: http://python.org/psf/codeofconduct/
