[
https://issues.apache.org/jira/browse/SPARK-61162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61162:
------------------------------------
Assignee: Hyukjin Kwon
> Share the streaming query listener bus of a session, post the started events
> of queries, and give the listeners the session, in the Rust and Python
> clients, as PySpark does
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61162
> URL: https://issues.apache.org/jira/browse/SPARK-61162
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> PySpark Connect's spark.streams is one StreamingQueryManager of the session,
> whose StreamingQueryListenerBus holds the client-side listeners:
> - addListener gives the listener the session (listener.spark), and, for the
> first listener, returns once the server has added its listener
> - start() of a query posts its started event, which the server sends with the
> result of the start (query_started_event_json), to the listeners before it
> returns
> - removeListener of the last listener asks the server to remove its listener
> and waits for the events left
> In the Rust client, each spark.streams() / session.streams() is a new manager
> with a new listener bus of its own, and in the Python client spark.streams is
> a new object at each access:
> - onQueryStarted is never called: the client ignores query_started_event_json
> - listeners have no session (self.spark is None), so a listener that runs
> queries fails
> - a listener added through one spark.streams is removed through another only
> because removing from an empty bus asks the server to remove the listener of
> the session, which also cuts the other listeners off
> - addListener returns before the server has added its listener, so a query
> started right after misses its events
> - spark.streams == spark.streams is False, and StreamingQueryManager has no
> _sqlb (the bus of the listeners, which Apache Spark's tests read)
> Six tests of Apache Spark's test_parity_listener.py and
> test_query_manager_no_recreation of test_parity_streaming.py fail on this
> client.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
