Hyukjin Kwon created SPARK-61117:
------------------------------------
Summary: Call the hooks of SparkSession.Hook before each
execution, in the Python client as PySpark does
Key: SPARK-61117
URL: https://issues.apache.org/jira/browse/SPARK-61117
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark 4.3.0, {{SparkSession.Builder._registerHook(factory)}} registers a
factory of {{SparkSession.Hook}}. The session calls each factory with itself.
Before it sends each {{ExecutePlanRequest}}, the client calls the
{{on_execute_plan}} of each hook with the request, and sends the request that
the hook returns, with the operation ID of the original.
The Python client accepts {{_registerHook}} but calls neither the factories nor
the hooks, so a hook that adds tags to a request or changes its plan has no
effect.
The fix is to call the factories when the session is created, and the hooks
before each execution of the Rust transport, with the request as a message of
the generated {{pyspark.sql.connect.proto}} (SPARK-61111).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
