[issue46391] Library multiprocess leaks named resources.

[XD Trol]

New submission from XD Trol :

Repo is the standard tool that Google uses to build Android, Chrome OS, 
Chromium, etc, written in Python. 

Many Repo users have encountered resource leak warnings with Python 3.9. 
https://bugs.chromium.org/p/gerrit/issues/detail?id=14934&q=component%3Arepo&can=5

I did some work and found that the problem is not caused by the code of Repo, 
but a bug of the Python library, multiprocess.

To make it simple, the Python script below leaks named resource even when 
exiting normally. (And be unlinked by resource_tracker with a warning. )

```
import multiprocessing as mp

global_resource = mp.Semaphore()

def submain(): pass

if __name__ == '__main__':
p = mp.Process(target=submain)
p.start()
p.join()
```

Tested on macOS with Python 3.9.7
> python test.py
resource_tracker: There appear to be 1 leaked semaphore objects to clean up at 
shutdown.

This bug will 100% reproduce when then main module uses named resources as 
global variables and uses `spawn` context, which is the case of Repo on macOS.

This is caused by multiprocess::BaseProcess::_bootstrap.

When a new process is started with multiprocessing.Process.start() in `spawn` 
context.
1. The main module is reloaded in the subprocess (for pickle) in 
multiprocessing::spawn::_main.
2. Named resources (such as the semaphore above) in the main module resister 
their _cleanup into multiprocessing::util::_finalizer_registry, which unlink 
themselves.
3. multiprocess::BaseProcess::_bootstrap then clears _finalizer_registry.

When a subprocess is spawned, it is no need to clear util::_finalizer_registry 
(and no need to call util::_run_after_forkers). Disable clearing 
_finalizer_registry (and disable call to _run_after_forkers) should fix this 
bug without breaking anything else.

And I uploaded a MR.

--
components: Library (Lib)
messages: 410654
nosy: milestonejxd
priority: normal
severity: normal
status: open
title: Library multiprocess leaks named resources.
type: resource usage
versions: Python 3.9

___
Python tracker 
<https://bugs.python.org/issue46391>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46391] Library multiprocess leaks named resources.

[XD Trol]

Change by XD Trol :


--
keywords: +patch
pull_requests: +28821
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/30617

___
Python tracker 
<https://bugs.python.org/issue46391>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46391] Library multiprocess leaks named resources.

[XD Trol]

XD Trol  added the comment:

loop file owner.

--
nosy: +benjamin.peterson, sbt

___
Python tracker 
<https://bugs.python.org/issue46391>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46391] Library multiprocess leaks named resources.

[XD Trol]

Change by XD Trol :


--
nosy: +yselivanov

___
Python tracker 
<https://bugs.python.org/issue46391>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46391] Library multiprocess leaks named resources.

[XD Trol]

XD Trol  added the comment:

update,

Confirmed to affect all versions above 3.8+.

It's really annoying. 

Command line tools like Repo get a warning after every (success) command. 
And mess up with shell.

---

$ repo status .
project ksbox/  branch dev
$ 
/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216:
 UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects 
to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

$
$
$
$ repo sync .
Fetching: 100% (1/1), done in 1.184s
Garbage collecting: 100% (1/1), done in 0.013s
repo sync has finished successfully.
$ 
/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216:
 UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects 
to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

$
$
$
$ repo status .
project ksbox/  branch dev
$ 
/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216:
 UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects 
to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

--
versions: +Python 3.10, Python 3.11, Python 3.8
Added file: https://bugs.python.org/file50580/screen.png

___
Python tracker 
<https://bugs.python.org/issue46391>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com