[issue35279] asyncio uses too many threads by default

Change by Inada Naoki : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___ P

[issue35279] asyncio uses too many threads by default

Inada Naoki added the comment: New changeset 9a7e5b1b42abcedb895b1ce49d83fe067d01835c by Inada Naoki in branch 'master': bpo-35279: reduce default max_workers of ThreadPoolExecutor (GH-13618) https://github.com/python/cpython/commit/9a7e5b1b42abcedb895b1ce49d83fe067d01835c -- _

[issue35279] asyncio uses too many threads by default

Change by Inada Naoki : -- keywords: +patch pull_requests: +13520 stage: -> patch review pull_request: https://github.com/python/cpython/pull/13618 ___ Python tracker ___

[issue35279] asyncio uses too many threads by default

Andrew Svetlov added the comment: > how about min(32, cpu_count+4)? I think it produces reasonable good numbers for any CPU count. Do you have time for preparing a pull request? -- ___ Python tracker _

[issue35279] asyncio uses too many threads by default

Inada Naoki added the comment: > If you want to limit to 16-20 that may be ok but `cpu_count + 4` doesn't work > in this case. On cloud servers, I see 128 or even more cores very often. > 160+4 is not that you want to propose, sure. I proposed cpu_count + 4 because #24882 almost fixed the p

[issue35279] asyncio uses too many threads by default

Andrew Svetlov added the comment: I'm ok with changing the default threads number limit. Not sure about numbers. If you want to limit to 16-20 that may be ok but `cpu_count + 4` doesn't work in this case. On cloud servers, I see 128 or even more cores very often. 160+4 is not that you want to

[issue35279] asyncio uses too many threads by default

Inada Naoki added the comment: Current default value is decided in here: https://bugs.python.org/review/21527/#ps11902 It seems there were no strong reason for current cpu_count * 5. I think cpu_count + 4 is better default value, because: * When people are using threadpool for CPU heavy job w

[issue35279] asyncio uses too many threads by default

Andrew Svetlov added the comment: asyncio uses bare concurrent.futures.ThreadPoolExecutor. Tha question is: should asyncio reduce the number of threads in the pool or concurrent.futures should change the default value? -- ___ Python tracker

[issue35279] asyncio uses too many threads by default

Inada Naoki added the comment: node.js default threadpool size is 4 regardless number of cores. https://nodejs.org/api/cli.html#cli_uv_threadpool_size_size Since we has GIL, I think fixed-size pool is better idea. -- ___ Python tracker

[issue35279] asyncio uses too many threads by default

Laurie Opperman added the comment: What about making it dependant on memory as well as logical processor count: `n_workers = min(RAM_GB / some_number, N_CORES * 5)` -- nosy: +Epic_Wink ___ Python tracker __

[issue35279] asyncio uses too many threads by default

Change by Inada Naoki : -- nosy: +inada.naoki ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.py

[issue35279] asyncio uses too many threads by default

New submission from Vojtěch Boček : By default, asyncio spawns as many as os.cpu_count() * 5 threads to run I/O on. When combined with beefy machines (e.g. kubernetes servers) with, says, 56 cores, it results in very high memory usage. This is amplified by the fact that the `concurrent.futures