At work we faced a problem of long running python code. Our case was a short
task, but huge count of iterations. Something like:
for x in data_list:
# do 1ms non-io pure python task
So we block loop for more than 100ms, or even 1000ms. The first naive solution
was "move this to thread" so python will switch thread context and asyncio loop
will not be blocked, but this raised two problems:
* If we do asyncio things in our task (create `Future` in our case), then we
need to pass loop explicitly and use `call_soon_threadsafe`
* We still saw asyncio warnings about blocking the loop. Not sure why, but
maybe because of GIL was not released when main/asyncio thread became active.
We endend up with wrapper for iterable, which switch «asyncio context» via
`asyncio.sleep(0)` (since we know that sleep(0) have special code, which just
switches context) by time or by count. Here is our code:
async def iterate_non_blocking(iterable, context_switch_interval=0.01,
context_switch_count=None):
last_context_switch_time = time.perf_counter()
for i, value in enumerate(iterable, start=1):
yield value
switch_context_by_interval = context_switch_interval and \
(time.perf_counter() - last_context_switch_time) >=
context_switch_interval
switch_context_by_count = context_switch_count and i %
context_switch_count == 0
if switch_context_by_interval or switch_context_by_count:
await asyncio.sleep(0)
last_context_switch_time = time.perf_counter()
I'm not sure if this is a good approach, but this solves all our problems for
this kind of blocking cases. Here comes discussable things:
* Is there a better approach for such cases?
* Should this be a part of asyncio?
* Should this be a part of documentation as recipe?
_______________________________________________
Python-ideas mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/EZWJS6OTBDG7LCX2WECDMPET6I5BMGDU/
Code of Conduct: http://python.org/psf/codeofconduct/
