STINNER Victor <[email protected]> added the comment:
> And setup.py does use use multiple processes via a thread pool, where each
> thread calls fork(), if it detects that "make" was invoked with the parallel
> (-j) option.
Oh ok, now I get it :-) setup.py uses distutils to build extensions and
distutils.command.build_ext uses concurrent.futures for parallelism:
if self.parallel:
self._build_extensions_parallel()
else:
self._build_extensions_serial()
with:
def _build_extensions_parallel(self):
workers = self.parallel
if self.parallel is True:
workers = os.cpu_count() # may return None
try:
from concurrent.futures import ThreadPoolExecutor
except ImportError:
workers = None
if workers is None:
self._build_extensions_serial()
return
with ThreadPoolExecutor(max_workers=workers) as executor:
futures = [executor.submit(self.build_extension, ext)
for ext in self.extensions]
for ext, fut in zip(self.extensions, futures):
with self._filter_build_errors(ext):
fut.result()
The problem is not concurrent.futures but the job submitted to the executor:
build_extension() uses distutils.spawn() which is unsafe.
distutils.spawn() is the root issue if I understood correctly. It must be
rewritten with subprocess.
----------
_______________________________________
Python tracker <[email protected]>
<https://bugs.python.org/issue39763>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
