New submission from XD Trol <milestone...@gmail.com>:
Repo is the standard tool that Google uses to build Android, Chrome OS, Chromium, etc, written in Python. Many Repo users have encountered resource leak warnings with Python 3.9. https://bugs.chromium.org/p/gerrit/issues/detail?id=14934&q=component%3Arepo&can=5 I did some work and found that the problem is not caused by the code of Repo, but a bug of the Python library, multiprocess. To make it simple, the Python script below leaks named resource even when exiting normally. (And be unlinked by resource_tracker with a warning. ) ``` import multiprocessing as mp global_resource = mp.Semaphore() def submain(): pass if __name__ == '__main__': p = mp.Process(target=submain) p.start() p.join() ``` Tested on macOS with Python 3.9.7 > python test.py resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown. This bug will 100% reproduce when then main module uses named resources as global variables and uses `spawn` context, which is the case of Repo on macOS. This is caused by multiprocess::BaseProcess::_bootstrap. When a new process is started with multiprocessing.Process.start() in `spawn` context. 1. The main module is reloaded in the subprocess (for pickle) in multiprocessing::spawn::_main. 2. Named resources (such as the semaphore above) in the main module resister their _cleanup into multiprocessing::util::_finalizer_registry, which unlink themselves. 3. multiprocess::BaseProcess::_bootstrap then clears _finalizer_registry. When a subprocess is spawned, it is no need to clear util::_finalizer_registry (and no need to call util::_run_after_forkers). Disable clearing _finalizer_registry (and disable call to _run_after_forkers) should fix this bug without breaking anything else. And I uploaded a MR. ---------- components: Library (Lib) messages: 410654 nosy: milestonejxd priority: normal severity: normal status: open title: Library multiprocess leaks named resources. type: resource usage versions: Python 3.9 _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue46391> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
