Hyukjin Kwon created SPARK-61166:
------------------------------------
Summary: Make the threads of inheritable_thread_target share the
copy of the tags of the session, in the Rust and Python clients, as PySpark does
Key: SPARK-61166
URL: https://issues.apache.org/jira/browse/SPARK-61166
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark Connect's inheritable_thread_target(session) copies the thread_local of
the client, of the tags of the thread, when it decorates a function, and gives
every thread that runs the function that one copy: the threads share it. So a
thread that the function starts from another thread of the function sees the
tags that that thread added, and the tags of the decorating thread stay its own.
The Python client copies the tags into each thread instead, so each thread
starts from the tags of the decorating thread and the threads do not share the
tags that they add. Apache Spark's test_inheritable_tags_with_deco of
test_parity_job_cancellation.py fails on this client (the nested thread lacks
the tag "b" of the thread that started it).
The thread-local user context extensions, which PySpark's decorator copies in
the same thread_local, are not propagated by the client's decorator at all;
that is not changed here.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
