Asyncio tasks getting cancelled
Hi all, I'm having trouble with asyncio. Apparently tasks (asyncio.create_task) are not kept referenced by asyncio itself, causing the task to be cancelled when the creating function finishes (and noone is awaiting the corresponding futue). Am I doing something wrong or is this expected behavior? The code sample I tried: > import asyncio > > async def foobar(): > print(1) > await asyncio.sleep(1) > print(2) > > async def main(): > asyncio.create_task(foobar()) > #await asyncio.sleep(2) > > loop = asyncio.get_event_loop() > asyncio.run(main()) > loop.run_forever() The sample does print "2" only when uncommenting the asyncio.sleep(2) line. Greetings. -- https://mail.python.org/mailman/listinfo/python-list
Re: Asyncio tasks getting cancelled
This weird mixing was actually a side effect of me quickly coming up with a small example, sorry for the confusion. I just saw, actually using the same loop gets rid of the behavior in this case and now I'm not sure about my assertions any more. Yet it still looks like asyncio doen'st keep strong references. Hmm. The original problem prompting my question is here: https://github.com/ldo/dbussy/issues/13 Also this is a python 3.7.1 on a current Fedora 29. > > Usually, when you're already running into a coroutine, you're using > > "await other_coroutine()" instead of > > "asyncio.create_task(other_coroutine())". > > Which is not accurate. What Ian said is accurate. One indeed may need to > schedule a coroutine in some situation. I just assumed that's wasn't > what the OP intended to do given the name of the "main" function. > > I also said: > > > But anyway, I highly recommend you to use the "await other_coroutine()" > > syntax I talked about earlier. It may even fix the issue (90% chance). > > This should indeed fix the issue, but this is definitely not what one is > looking for if one really want to _schedule_ a coroutine. Which is what I want in this case. Scheduling a new (long-running) task as a side effect, but returning early oneself. The new task can't be awaited right there, because the creating one should return already. > If the goal here is for the task created by main() to complete before > the loop exits, then main() should await it, and not just create it > without awaiting it. So if this happens somewhere deep in the hirarchy of your application you would need some mechanism to pass the created tasks back up the chain to the main function? Greetings. -- https://mail.python.org/mailman/listinfo/python-list
Re: Asyncio tasks getting cancelled
On Mon, Nov 05, 2018 at 01:57:56PM -0700, Ian Kelly wrote: > > Which is what I want in this case. Scheduling a new (long-running) task > > as a side effect, but returning early oneself. The new task can't be > > awaited right there, because the creating one should return already. > > If you want to do this in the asyncio.run main coroutine, then that > seems like a problematic design. Once the main coroutine returns, the > event loop should be considered no longer running, and any still > pending callbacks or futures won't resolve. This is only true for the small example I provided. In the actual code this is somewhere deep in the hirarchy. > > > If the goal here is for the task created by main() to complete before > > > the loop exits, then main() should await it, and not just create it > > > without awaiting it. > > > > So if this happens somewhere deep in the hirarchy of your application > > you would need some mechanism to pass the created tasks back up the > > chain to the main function? > > I haven't used asyncio.run yet myself, so take all this with a grain > of salt, but it seems to me that anything that you want to resolve > before the event loop terminates should be awaited either directly or > indirectly by the main coroutine. From the documentation: > > """ > This function always creates a new event loop and closes it at the > end. It should be used as a main entry point for asyncio programs, and > should ideally only be called once. > """ > > So I think part of the idea with this is that the asyncio.run main > coroutine is considered the main function of your async app. Once it > returns, the program should be effectively done. For example, maybe > the main coroutine spins up a web server and returns when the web > server shuts down. Again sorry for the confusion, but I don't think this is an issue with restarting loops, as this isn't happening in my application. For context: https://github.com/ldo/dbussy/issues/13 https://gist.github.com/tu500/3232fe03bd1d85b1529c558f920b8e43 It really feels like asyncio is loosing strong references to scheduled tasks, as excplicitly keeping them around helps. Also, the error messages I'm getting are the ones from here: https://github.com/python/cpython/blob/16c8a53490a22bd4fcde2efaf4694dd06ded882b/Lib/asyncio/tasks.py#L145 Which indicates that the tasks actually weren't even started at all? > If that doesn't suit your program, for instance there's no core task > to await, but you want to schedule a lot of things that need to > resolve and that the main coroutine has no way to know about, then it > may be the case that asyncio.run is not right for your use case and > you should use loop.run_forever() instead. You'll still need some > criteria for figuring out when to exit though, and it seems to me that > whatever that is you could just bundle it up in a coroutine and await > it from main. Though not really related with my actual problem, so getting off topic, but I can imagine an architecture where that would be "There aren't any running tasks any more." or even "Never." Also, I may be overlooking things, but I haven't found a way to add a task before calling run_forever(), as asyncio will then say the loop isn't running yet. So I'm not sure how you would jumpstart in that case. -- https://mail.python.org/mailman/listinfo/python-list
Re: Asyncio tasks getting cancelled
On Tue, Nov 06, 2018 at 12:45:03AM +0100, i...@koeln.ccc.de wrote: > Also, I may be overlooking things, but I haven't found a way to add a > task before calling run_forever(), as asyncio will then say the loop > isn't running yet. So I'm not sure how you would jumpstart in that case. Ok, I was confused there. It doesn't work by using asyncio.create_task but it will with loop.create_task. Sorry for the noise. -- https://mail.python.org/mailman/listinfo/python-list
Re: Asyncio tasks getting cancelled
On Mon, Nov 05, 2018 at 07:15:04PM -0700, Ian Kelly wrote: > > For context: > > https://github.com/ldo/dbussy/issues/13 > > https://gist.github.com/tu500/3232fe03bd1d85b1529c558f920b8e43 > > > > It really feels like asyncio is loosing strong references to scheduled > > tasks, as excplicitly keeping them around helps. Also, the error > > messages I'm getting are the ones from here: > > https://github.com/python/cpython/blob/16c8a53490a22bd4fcde2efaf4694dd06ded882b/Lib/asyncio/tasks.py#L145 > > Which indicates that the tasks actually weren't even started at all? > > No, it indicates that it was cleaned up (likely because the program > exited) before the task completed. Which likely implies that the loop > exited without waiting for it. From the stack trace, you're using > loop.run_until_complete(main()). Like asyncio.run, that only runs > until the specific thing you pass it completes, which might wait on > other things or might not. Anything else that's still pending is going > to be left up in the air unless you subsequently restart the same > event loop. What I meant was, the error message is specific to futures in the 'PENDING' state. Which should be set to 'RUNNING' before any actions occur. So it appears the tasks weren't started at all. Also the cleanup happens instantly, ie before the loop exits (as the main loop has an asyncio.sleep timeout. Printing a message also clearly shows this happens way before main returns. -- https://mail.python.org/mailman/listinfo/python-list
Re: Asyncio tasks getting cancelled
Sorry for the latency. On Tue, Nov 06, 2018 at 03:39:24PM -0700, Ian Kelly wrote: > > What I meant was, the error message is specific to futures in the > > 'PENDING' state. Which should be set to 'RUNNING' before any actions > > occur. So it appears the tasks weren't started at all. > > Ah. I don't think asyncio uses a RUNNING state. There's nothing about > it in the docs; tasks are either done or they're not: > https://docs.python.org/3/library/asyncio-task.html#task-object > > And inspecting the current task also shows PENDING: > > py> from asyncio import * > py> async def main(): > ... print(current_task()._state) > ... > py> run(main()) > PENDING Yes, you're right. I was confused and looked at futures, not asyncio.futures. :/ > > Also the cleanup happens instantly, ie before the loop exits (as the > > main loop has an asyncio.sleep timeout. Printing a message also clearly > > shows this happens way before main returns. > > Okay, that wasn't clear from the log you posted. It's not obvious to > me why that would be happening. although it does sound like the tasks > are getting created and then immediately garbage-collected for some > reason. It could be a bug in asyncio, or it could be a bug in the way > the tasks are created, e.g. on a separate event loop that gets dropped > on the floor. Yes, I should have mentioned that earlier, sorry. Hmm, I guess this will need some debugging incorporating the asyncio module. Oh well, let's see if I come around to that. Anyway thanks for the input. -- https://mail.python.org/mailman/listinfo/python-list
Re: Are all items in list the same?
You might do something like if len(a) == 0 or all(i == a[0] for i in a[1:]): This should be linear complexity and short circuiting and in general it doesn't get much better than this. Though I wouldn't bet there isn't a better (faster/clearer/more readable) solution. On Mon, Jan 07, 2019 at 05:14:14PM -0700, Bob van der Poel wrote: > I need to see if all the items in my list are the same. I was using set() > for this, but that doesn't work if items are themselves lists. So, assuming > that a is a list of some things, the best I've been able to come up with it: > > if a.count( targ ) == len(a): > > I'm somewhat afraid that this won't scale all that well. Am I missing > something? -- https://mail.python.org/mailman/listinfo/python-list
Re: Are all items in list the same?
On Tue, Jan 08, 2019 at 01:05:09PM +1100, Chris Angelico wrote: > No, because the iteration would never happen. If the list is empty, > a[1:] is also empty, and i==a[0] will never be evaluated. So it is > safe. (But I agree that it's not instantly obvious.) Oh yeah, you're right. I overthought it too much. :) -- https://mail.python.org/mailman/listinfo/python-list
