[issue36084] Threading: add builtin TID attribute to Thread objects

[Jake Tesler]

Jake Tesler  added the comment:

I have encountered a minor bug with this new feature. The bug occurs when 
creating a new multiprocessing.Process object on Unix (or on any platform where 
the multiprocessing start_method is 'fork' or 'forkserver').

When creating a new process via fork, the Native ID in the new MainThread is 
incorrect. The new forked process' threading.MainThread object inherits the 
Native ID from the parent process' MainThread instead of capturing/updating its 
own (new) Native ID.

See the following snippet:

>>> import threading, multiprocessing
>>> multiprocessing.set_start_method('fork') # or 'forkserver'
>>> def proc(): print(threading.get_native_id(), 
>>> threading.main_thread().native_id) # get_native_id(), mainthread.native_id
>>> proc()
22605 22605 # get_native_id(), mainthread.native_id
>>> p = multiprocessing.Process(target=proc)
>>> p.start()
22648 22605 # get_native_id(), mainthread.native_id
>>>
>>> def update(): threading.main_thread()._set_native_id()
>>> def print_and_update(): proc(); update(); proc()
>>> print_and_update()
22605 22605 # get_native_id(), mainthread.native_id
22605 22605 
>>> p2=multiprocessing.Process(target=print_and_update); p2.start()
22724 22605 # get_native_id(), mainthread.native_id
22724 22724
>>> print_and_update()
22605 22605 # get_native_id(), mainthread.native_id
22605 22605

As you can see, the new Process object's MainThread.native_id attribute matches 
that of the MainThread of its parent process. 

Unfortunately, I'm not too familiar with the underlying mechanisms that 
Multiprocessing uses to create forked processes. 
I believe this behavior occurs because (AFAIK) a forked multiprocessing.Process 
copies the MainThread object from its parent process, rather than 
reinitializing a new one. Looking further into the multiprocessing code, it 
appears the right spot to fix this would be in the 
multiprocessing.Process.bootstrap() function. 

I've created a branch containing a working fix - I'm also open to suggestions 
of how a fix might otherwise be implemented. 
If it looks correct I'll create a PR against the CPython 3.8 branch. 

See the branch here: https://github.com/jaketesler/cpython/tree/fix-mp-native-id

Thanks all!
-Jake

--
status: closed -> open

___
Python tracker 
<https://bugs.python.org/issue36084>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36084] Threading: add builtin TID attribute to Thread objects

[Jake Tesler]

Change by Jake Tesler :


--
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue36084>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38707] Multiprocessing: bug with Native ID for threading.mainthread()

[Jake Tesler]

New submission from Jake Tesler :

I have encountered a minor bug with the new `threading.get_native_id()` 
featureset in Python 3.8. The bug occurs when creating a new 
multiprocessing.Process object on Unix (or on any platform where the 
multiprocessing start_method is 'fork' or 'forkserver').

When creating a new process via fork, the Native ID in the new MainThread is 
incorrect. The new forked process' threading.MainThread object inherits the 
Native ID from the parent process' MainThread instead of capturing/updating its 
own (new) Native ID.

See the following snippet:

>>> import threading, multiprocessing
>>> multiprocessing.set_start_method('fork') # or 'forkserver'
>>> def proc(): print(threading.get_native_id(), 
>>> threading.main_thread().native_id) # get_native_id(), mainthread.native_id
>>> proc()
22605 22605 # get_native_id(), mainthread.native_id
>>> p = multiprocessing.Process(target=proc)
>>> p.start()
22648 22605 # get_native_id(), mainthread.native_id
>>>
>>> def update(): threading.main_thread()._set_native_id()
>>> def print_and_update(): proc(); update(); proc()
>>> print_and_update()
22605 22605 # get_native_id(), mainthread.native_id
22605 22605 
>>> p2=multiprocessing.Process(target=print_and_update); p2.start()
22724 22605 # get_native_id(), mainthread.native_id
22724 22724
>>> print_and_update()
22605 22605 # get_native_id(), mainthread.native_id
22605 22605

As you can see, the new Process object's MainThread.native_id attribute matches 
that of the MainThread of its parent process. 

Unfortunately, I'm not too familiar with the underlying mechanisms that 
Multiprocessing uses to create forked processes. 
I believe this behavior occurs because (AFAIK) a forked multiprocessing.Process 
copies the MainThread object from its parent process, rather than 
reinitializing a new one. Looking further into the multiprocessing code, it 
appears the right spot to fix this would be in the 
multiprocessing.Process.bootstrap() function. 

I've created a branch containing a working fix - I'm also open to suggestions 
of how a fix might otherwise be implemented. 
If it looks correct I'll create a PR against the CPython 3.8 branch. 

See the branch here: https://github.com/jaketesler/cpython/tree/fix-mp-native-id

Thanks all!
-Jake

--
components: Library (Lib)
messages: 356070
nosy: jaketesler, vstinner
priority: normal
severity: normal
status: open
title: Multiprocessing: bug with Native ID for threading.mainthread()
type: behavior
versions: Python 3.8

___
Python tracker 
<https://bugs.python.org/issue38707>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38707] Multiprocessing: bug with Native ID for threading.mainthread()

[Jake Tesler]

Change by Jake Tesler :


--
nosy: +pitrou

___
Python tracker 
<https://bugs.python.org/issue38707>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38707] Multiprocessing: bug with Native ID for threading.mainthread()

[Jake Tesler]

Change by Jake Tesler :


--
keywords: +patch
pull_requests: +16596
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17088

___
Python tracker 
<https://bugs.python.org/issue38707>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38707] Multiprocessing: bug with Native ID for threading.mainthread()

[Jake Tesler]

Jake Tesler  added the comment:

@vstinner PR created :)
https://github.com/python/cpython/pull/17088

--

___
Python tracker 
<https://bugs.python.org/issue38707>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38707] Multiprocessing: bug with Native ID for threading.mainthread()

[Jake Tesler]

Jake Tesler  added the comment:

PR was updated with tests and is ready for core developer review and then the 
merge to cpython:master. After that (if I understand correctly) a backport will 
automatically get picked into the 3.8 branch if there aren't any conflicts.

--

___
Python tracker 
<https://bugs.python.org/issue38707>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36084] Threading: add builtin TID attribute to Thread objects

[Jake Tesler]

Jake Tesler  added the comment:

I will implement these changes - let’s try to hit that end-of-month target!

--

___
Python tracker 
<https://bugs.python.org/issue36084>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36084] Threading: add builtin TID attribute to Thread objects

[Jake Tesler]

Change by Jake Tesler :


--
pull_requests: +13373

___
Python tracker 
<https://bugs.python.org/issue36084>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36084] Threading: add builtin TID attribute to Thread objects

[Jake Tesler]

Jake Tesler  added the comment:

New PR created with requested edits. :)

--

___
Python tracker 
<https://bugs.python.org/issue36084>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36084] Threading: add builtin TID attribute to Thread objects

[Jake Tesler]

Jake Tesler  added the comment:

Victor Stinner: would you mind taking a look at the new PR? Is this more along 
the lines of what you had in mind?

--

___
Python tracker 
<https://bugs.python.org/issue36084>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36084] Threading: add builtin TID attribute to Thread objects

[Jake Tesler]

Jake Tesler  added the comment:

In general, I’ve concluded most ‘editions’ of pthread_self() are not the same 
value as this feature aims to implement. I’m not familiar enough with AIX to be 
certain about that platform, though. 

If there’s an equivalent function in AIX to capture the actual integral thread 
ID in the same 3-line way as Linux, FreeBSD, et al., are implemented (the big 
block in PyThread_get_thread_native_id), then it’s worth including. If the 
mechanism required is a more complex one than just adding a few lines, then it 
might be best left for a new PR. :)

--

___
Python tracker 
<https://bugs.python.org/issue36084>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36084] Threading: add builtin TID attribute to Thread objects

[Jake Tesler]

Jake Tesler  added the comment:

I will look into whether adding thread_self() for AIX would be simple enough 
for this PR.

--

___
Python tracker 
<https://bugs.python.org/issue36084>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36084] Threading: add builtin TID attribute to Thread objects

[Jake Tesler]

Jake Tesler  added the comment:

Victor – the return value of _start_new_thread is the the `ident` parameter, 
and its not the same as the native id.

See here: https://github.com/python/cpython/pull/11993#issuecomment-491544908

--

___
Python tracker 
<https://bugs.python.org/issue36084>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36084] Threading: add builtin TID attribute to Thread objects

[Jake Tesler]

Jake Tesler  added the comment:

Michael Felt -
If you would like some help with adding/building AIX support for this 
functionality, tag me, I'd be glad to help out! :)

--

___
Python tracker 
<https://bugs.python.org/issue36084>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36084] Threading: add builtin TID attribute to Thread objects

[Jake Tesler]

Change by Jake Tesler :


--
pull_requests: +14636
pull_request: https://github.com/python/cpython/pull/14845

___
Python tracker 
<https://bugs.python.org/issue36084>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36084] Threading: add builtin TID attribute to Thread objects

[Jake Tesler]

New submission from Jake Tesler :

This functionality adds a native Thread ID to threading.Thread objects. This ID 
(TID), similar to the PID of a process, is assigned by the OS (kernel) and is 
generally used for externally monitoring resources consumed by the running 
thread (or process).
This does not replace the `ident` attribute within Thread objects, which is 
assigned by the Python interpreter and is guaranteed as unique for the lifetime 
of the Python instance.

--
messages: 336348
nosy: Jake Tesler
priority: normal
severity: normal
status: open
title: Threading: add builtin TID attribute to Thread objects
type: enhancement
versions: Python 3.8

___
Python tracker 
<https://bugs.python.org/issue36084>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36084] Threading: add builtin TID attribute to Thread objects

[Jake Tesler]

Jake Tesler  added the comment:

*bump*

Could someone look into reviewing this bug/PR? Thanks!

--

___
Python tracker 
<https://bugs.python.org/issue36084>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36084] Threading: add builtin TID attribute to Thread objects

[Jake Tesler]

Jake Tesler  added the comment:

The feature is supported on Windows: the file supporting Windows threading is 
`thread_nt.h`, not `thread_pthread.h` since Windows doesn't use POSIX-style 
threads.

Also it is different from threading.get_ident() - ident is a Python-issued 
unique identifier, TID is issued by the OS and is tracable system-wide.

--

___
Python tracker 
<https://bugs.python.org/issue36084>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36084] Threading: add builtin TID attribute to Thread objects

[Jake Tesler]

Jake Tesler  added the comment:

So where do we go from here?

--

___
Python tracker 
<https://bugs.python.org/issue36084>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com