Despite my concerns over code for an implementation on my previous e-mail,
it turns out that simply iterating in an `async for` loop won't yield
to the asyncio-loop. An explicit "await" inside the async-generator
is needed for that.
That makes factoring-out the code presented in the first e-mail
in this thread somewhat trivial - and it will read something like:
```
async def switch():
await asyncio.sleep(0)
async def iterate_non_blocking(iterator, timeout=None, iterations=None):
if timeout is None and iterations is None:
timeout = 0.1
ts = time.time()
counter = 0
for item in iterator:
yield item
counter += 1
if iterations and (counter >= iterations) or timeout and
time.time() - ts >= timeout:
await switch()
counter = 0
ts = time.time()
```
Which can then be used like:
```
async for i in iterate_non_blocking(range(steps), iterations=30):
...
```
I've created a gist with this code at
https://gist.github.com/jsbueno/ae6001c55ee3ff001fb7c152b1f109b2
And if people agree it is imporant enough, I am ok with creating a
full-package with this code,
or help adding it to the stdlib.
On Fri, 14 Jun 2019 at 11:58, Joao S. O. Bueno <[email protected]>
wrote:
> So -
>
> Now thinking on the problem as a whole -
> I think maybe a good way to address this is to put the logic on
> "counting N interations or X time and allowing switch" - the logic you
> had to explicitly mingle in your code in the first example, in
> a function that could wrap the iterator of the `for` loop.
>
> However, that idea would need a _synchronous_ call to the loop
> to force an async context switch - I don't know if
> there is such a call (even an internal one). Actually
> I don't know it that is possible - but I can't think of other way of
> factoring it out without explicitly triggering an await.
>
> The switcher would then be used just as we use `enumerate` -
> think of something along:
>
>
> for context_switch, data_set in async_switcher(data_sets, timeout=100):
> for record in context_switch(data_set):
> # the call to __iter__ here would include the logic to decide
> whether to switch,
> ...
>
> And for a single (non-nested) loop, either a separate call or:
>
> for data_set in next(async_switcher(data_sets, timeout=100))[0]:
> ...
>
> All in all: if there is a valid way to force the async-context switch in
> an
> sync-call to this wrapper object, it is possible to create a small package
> that would have this feature and be easy to use.
> (And then, we discuss further down if this is stdlib worthy)
>
>
>
>
>
> On Fri, 14 Jun 2019 at 09:45, Nikita Melentev <[email protected]>
> wrote:
>
>> The problem here is that even if I have a coroutine all code between
>> «awaits» is blocking.
>>
>> ``` python
>> async def foo():
>> data = await connection.get() # it is ok, loop handling request, we
>> waiting
>> # from here
>> for item in data: # this is 10 ** 6 len
>> do_sync_jon(item) # this took 1ms
>> # to here we are blocking loop for 1 second
>> await something_next()
>> ```
>> _______________________________________________
>> Python-ideas mailing list -- [email protected]
>> To unsubscribe send an email to [email protected]
>> https://mail.python.org/mailman3/lists/python-ideas.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/[email protected]/message/BLLU545Y2FLACLMHC6OVXGJ5YUF66E4K/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
_______________________________________________
Python-ideas mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/EYDG5HHRM2GC3O67IRTWDXVDAWD4YIKB/
Code of Conduct: http://python.org/psf/codeofconduct/
