Hyukjin Kwon created SPARK-61153:
------------------------------------
Summary: Run the queries of progress handlers and hooks, and keep
an execution reattachable when a hook adds reattach options, as PySpark does
Key: SPARK-61153
URL: https://issues.apache.org/jira/browse/SPARK-61153
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
A query run inside a progress handler (registerProgressHandler) or inside
SparkSession.Hook.on_execute_plan panics with "Cannot start a runtime from
within a runtime", because the shared runtime's block_on is called from a
thread already in the runtime's context. Official PySpark runs the nested query.
A hook that appends a request option with reattachable = False makes every
execution fail with INVALID_HANDLE.OPERATION_NOT_FOUND after its rows are read.
The reattachable iterator appends its own reattachable = True only when no
option exists, while PySpark always appends it after the hooks ran.
Repro: a progress handler running s.range(4).count(), a hook running another
query, and a hook appending reattach_options.reattachable = False on
s.range(2).count().
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
