[Python-ideas] Re: PEP 572: Is it a good ideas that walrus operator can be used in assertions since it causes side effects?
Thank you for all your answers. They were all very instructive to me. However, after further investigation, I feel that I have made a mistake when putting this topic on the table. When walrus operator was discussed, it was proposed to restrict assignment expressions to if, elif, and while statements. That proposal was rejected. I realised now that that also applies, by analogy, to assert statement. So, sorry for taking this issue back. My apologies. I was not my intention to re-start any superseded debate. The walrus operator, as you know, has been —and will be— highly controversial and material on this matter is enormous. Reviewing all this material is a huge task that I should have done before speaking. Sorry again and thank you. ___ Python-ideas mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/MACCFXXNIL3EDUAFR7ZVGK36B4I6SATO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Asyncio Future.set_result_threadsafe
I agree that is counter to the basic philosophy, though there are potential use cases where it may be unavoidable to use threads. It seems the best solutions when it is then are to use run_in_executor, which automatically handles the waking up of the loop. While the Queue object doesn't work with threads, though there does seem to be a library janus which provides a thread-capbable queue object which could be another solution. -Original Message- From: Guido van Rossum To: Brian Allen Vanderburg II Cc: Python-Ideas Sent: Thu, Feb 13, 2020 1:18 am Subject: Re: [Python-ideas] Asyncio Future.set_result_threadsafe Sharing futures between threads like that goes counter to asyncio's basic philosophy (which is not to use threads :-). You already showed the solution: future._loop.call_soon_threadsafe(future.set_result, ...). If that's unacceptable for you, maybe you can wrap the future in a wrapper class that calls call_soon_threadsafe. On Wed, Feb 12, 2020 at 8:51 PM Brian Allen Vanderburg II via Python-ideas wrote: Currently asyncio.futures.Future.set_result will result in any callbacks being scheduled using loop.call_soon instead of loop.call_soon_threadsafe. However in situations where the future's result is set from a different thread, the loop might not wake up as a result of this if it is currently sleeping. def compute_something(...): loop = asyncio.get_event_loop() future = loop.create_future() manager.run_computation(future, ...) # add the computation to a thread that is running it return future async def compute_stuff(): result = await compute_something(...) loop = asyncio.get_event_loop() loop.run_until_complete(compute stuff()) loop.close() The reasoning behind it is after yielding the Future object, the ready list of the event loop is empty so when waiting for selectors the timeout value is None and it just waits. When the other thread calls set_result, the loop ready list will get updated, but it's still waiting on selectors. The other thread could call future._loop.call_soon_threadsafe(future.set_result, ...), which writes some bytes to a socket in order to wake the even loop. I'm aware there are other ways to do things like this (queues, executors, etc). It just seems that, regardless of any method used, the code holding the future with the purpose of setting a result on it should be able to be blind to other implementation details. future.set_result should be all it needs to know to call even if from another thread, or if not, a method like future.set_result_threadsafe, which would hide the details of the loop.call_soon/call_soon_threadsafe ___ Python-ideas mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/PLKYKFWM4EM7DHUHWOBDU2JSJM57X7GM/ Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido)Pronouns: he/him (why is my pronoun here?)___ Python-ideas mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/CWV6UQUN4LFZHQK53L463OY4D4EROFXS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: PEP 614: Relaxing Grammar Restrictions On Decorators
Thanks for all of your feedback. I've updated the PEP to explain the current restrictions up front, and removed the section covering the interaction between decorators and binary @ operations. My overall impression is that there were no issues with the substance of the PEP itself, just its presentation. If there are no major objections raised in the next few days, I'll bring the discussion over to Python-Dev. Brandt ___ Python-ideas mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/5FKT5GHP4FV5LBTQSKIY3FGDIWGLRPSZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Asyncio Future.set_result_threadsafe
> I agree that is counter to the basic philosophy, though there are potential use cases where it may be unavoidable to use threads. It seems the best solutions when it is then are to use run_in_executor, which automatically handles the waking up of the loop. While this doesn't provide an immediate solution, we're currently working on implementing an asyncio.ThreadPool for Python 3.9, which is made to be more self-contained (initialized and finalized within an async context manager using ``async with``) and with a higher level API compared to loop.run_in_executor(). In general, I'd recommend making use of either of those two compared to directly using threads in asyncio. That being said, if you need more fine-grained control, there's also one other alternative low-level construct/design pattern you can use for working with threads in asyncio that we use in the implementation of some coroutine methods, such as one that I recently added to 3.9 called loop.shutdown_default_executor(). Using a similar format to your above example: ``` def _do_complete_something(fut, loop, ...): try: # perform some action, optionally get a result to set to the future loop.call_soon_threadsafe(future.set_result, None) # replace None with actual result if needed except Exception as ex: loop.call_soon_threadsafe(future.exception, ex) async def compute_something(...): # In general, I'd recommend using get_running_loop() to get the event loop within a coroutine function. # It has more predictable behavior compared to get_event_loop(). (Python 3.7+) loop = asyncio.get_running_loop() fut = loop.create_future() th = threading.Thread(target=_do_compute_something, args=(fut,loop, ...)) try: # use "return await future" if the value is desired await future finally: th.join() async def compute_stuff(): result = await compute_something(...) asyncio.run(compute_stuff()) ``` Hopefully that helps a bit, or at least gives you some some ideas. On Thu, Feb 13, 2020 at 9:31 AM Brianvanderburg2 via Python-ideas < [email protected]> wrote: > I agree that is counter to the basic philosophy, though there are > potential use cases where it may be unavoidable to use threads. It seems > the best solutions when it is then are to use run_in_executor, which > automatically handles the waking up of the loop. While the Queue object > doesn't work with threads, though there does seem to be a library janus > which provides a thread-capbable queue object which could be another > solution. > > > -Original Message- > From: Guido van Rossum > To: Brian Allen Vanderburg II > Cc: Python-Ideas > Sent: Thu, Feb 13, 2020 1:18 am > Subject: Re: [Python-ideas] Asyncio Future.set_result_threadsafe > > Sharing futures between threads like that goes counter to asyncio's basic > philosophy (which is not to use threads :-). > > You already showed the solution: > future._loop.call_soon_threadsafe(future.set_result, ...). If that's > unacceptable for you, maybe you can wrap the future in a wrapper class that > calls call_soon_threadsafe. > > On Wed, Feb 12, 2020 at 8:51 PM Brian Allen Vanderburg II via Python-ideas > wrote: > > Currently asyncio.futures.Future.set_result will result in any callbacks > being scheduled using loop.call_soon instead of > loop.call_soon_threadsafe. However in situations where the future's > result is set from a different thread, the loop might not wake up as a > result of this if it is currently sleeping. > > > def compute_something(...): > loop = asyncio.get_event_loop() > future = loop.create_future() > manager.run_computation(future, ...) # add the computation to a > thread that is running it > return future > > async def compute_stuff(): > result = await compute_something(...) > > loop = asyncio.get_event_loop() > loop.run_until_complete(compute stuff()) > loop.close() > > > The reasoning behind it is after yielding the Future object, the ready > list of the event loop is empty so when waiting for selectors the > timeout value is None and it just waits. When the other thread calls > set_result, the loop ready list will get updated, but it's still waiting > on selectors. The other thread could call > future._loop.call_soon_threadsafe(future.set_result, ...), which writes > some bytes to a socket in order to wake the even loop. > > I'm aware there are other ways to do things like this (queues, > executors, etc). It just seems that, regardless of any method used, the > code holding the future with the purpose of setting a result on it > should be able to be blind to other implementation details. > future.set_result should be all it needs to know to call even if from > another thread, or if not, a method like future.set_result_threadsafe, > which would hide the details of the loop.call_soon/call_soon_threadsafe > > > ___ > Python-ideas mailing list -- [email protected]
[Python-ideas] Re: Asyncio Future.set_result_threadsafe
Oops, I forgot to start the thread in the above example for compute_something(), ``th.start()`` should go right after initializing the thread but before the try-finally block On Thu, Feb 13, 2020 at 6:15 PM Kyle Stanley wrote: > > I agree that is counter to the basic philosophy, though there are > potential use cases where it may be unavoidable to use threads. It seems > the best solutions when it is then are to use run_in_executor, which > automatically handles the waking up of the loop. > > While this doesn't provide an immediate solution, we're currently working > on implementing an asyncio.ThreadPool for Python 3.9, which is made to be > more self-contained (initialized and finalized within an async context > manager using ``async with``) and with a higher level API compared to > loop.run_in_executor(). In general, I'd recommend making use of either of > those two compared to directly using threads in asyncio. > > That being said, if you need more fine-grained control, there's also one > other alternative low-level construct/design pattern you can use for > working with threads in asyncio that we use in the implementation of some > coroutine methods, such as one that I recently added to 3.9 called > loop.shutdown_default_executor(). Using a similar format to your above > example: > > ``` > def _do_complete_something(fut, loop, ...): > try: > # perform some action, optionally get a result to set to the future > loop.call_soon_threadsafe(future.set_result, None) # replace None > with actual result if needed > except Exception as ex: > loop.call_soon_threadsafe(future.exception, ex) > > async def compute_something(...): > # In general, I'd recommend using get_running_loop() to get the event > loop within a coroutine function. > # It has more predictable behavior compared to get_event_loop(). > (Python 3.7+) > loop = asyncio.get_running_loop() > fut = loop.create_future() > th = threading.Thread(target=_do_compute_something, args=(fut,loop, > ...)) > try: > # use "return await future" if the value is desired > await future > finally: > th.join() > > async def compute_stuff(): > result = await compute_something(...) > > asyncio.run(compute_stuff()) > ``` > > Hopefully that helps a bit, or at least gives you some some ideas. > > On Thu, Feb 13, 2020 at 9:31 AM Brianvanderburg2 via Python-ideas < > [email protected]> wrote: > >> I agree that is counter to the basic philosophy, though there are >> potential use cases where it may be unavoidable to use threads. It seems >> the best solutions when it is then are to use run_in_executor, which >> automatically handles the waking up of the loop. While the Queue object >> doesn't work with threads, though there does seem to be a library janus >> which provides a thread-capbable queue object which could be another >> solution. >> >> >> -Original Message- >> From: Guido van Rossum >> To: Brian Allen Vanderburg II >> Cc: Python-Ideas >> Sent: Thu, Feb 13, 2020 1:18 am >> Subject: Re: [Python-ideas] Asyncio Future.set_result_threadsafe >> >> Sharing futures between threads like that goes counter to asyncio's basic >> philosophy (which is not to use threads :-). >> >> You already showed the solution: >> future._loop.call_soon_threadsafe(future.set_result, ...). If that's >> unacceptable for you, maybe you can wrap the future in a wrapper class that >> calls call_soon_threadsafe. >> >> On Wed, Feb 12, 2020 at 8:51 PM Brian Allen Vanderburg II via >> Python-ideas wrote: >> >> Currently asyncio.futures.Future.set_result will result in any callbacks >> being scheduled using loop.call_soon instead of >> loop.call_soon_threadsafe. However in situations where the future's >> result is set from a different thread, the loop might not wake up as a >> result of this if it is currently sleeping. >> >> >> def compute_something(...): >> loop = asyncio.get_event_loop() >> future = loop.create_future() >> manager.run_computation(future, ...) # add the computation to a >> thread that is running it >> return future >> >> async def compute_stuff(): >> result = await compute_something(...) >> >> loop = asyncio.get_event_loop() >> loop.run_until_complete(compute stuff()) >> loop.close() >> >> >> The reasoning behind it is after yielding the Future object, the ready >> list of the event loop is empty so when waiting for selectors the >> timeout value is None and it just waits. When the other thread calls >> set_result, the loop ready list will get updated, but it's still waiting >> on selectors. The other thread could call >> future._loop.call_soon_threadsafe(future.set_result, ...), which writes >> some bytes to a socket in order to wake the even loop. >> >> I'm aware there are other ways to do things like this (queues, >> executors, etc). It just seems that, regardless of any method used, the >> code holding the future with the purpose of setting a result on i
