[
https://issues.apache.org/jira/browse/SPARK-61097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-61097:
----------------------------
Description:
The session-shared manager and synchronous query-start callbacks landed in
SPARK-61162 (PR #353, commite8b1d42). This ticket now tracks the remaining
shutdown and failure behavior of that shared bus.
Current upstream holds an operation mutex while joining the listener worker. If
an external close or last-listener removal waits for that worker while its
callback calls a manager method, the two threads can deadlock. A failed manager
close also clears registrations and discards a live worker handle, preventing
removal retry. Quiet workers and progress trackers can retain native
session/bus state without local cancellation; stop after a failed server
release can leave the stream waiting.
The follow-up must avoid holding lifecycle locks across RPCs or joins, preserve
ordinary failed-removal/close retry state, and cancel local listener work
during terminal session shutdown even if server cleanup fails. It must retain
the newly added listener.spark, manager._sqlb, listener ordering, native
listener introspection, and synchronous start behavior.
Deterministic callback/RPC gates, failure retry, stop cleanup, and ownership
tests are being prepared. Source traces establish the issue; native validation
and independent review are pending.
was:
Repeated access to SparkSession.streams constructs independent
StreamingQueryManager instances and listener registries. For example,
spark.streams.addListener(listener) followed by spark.streams.close() closes a
different empty registry; removeListener also addresses a different registry.
Official PySpark caches the session manager. Preserve listener state for one
session while keeping new sessions isolated, and close the bus with the
session. Add regressions for repeated property access, listener removal and
cleanup.
Observed by source inspection of public apache/spark-connect-rust master
83d74462ec41012993176cf3af99017576688eb8. Runtime regression tests will
accompany the fix.
Summary: Avoid listener callback deadlocks and preserve retries during
streaming bus shutdown (was: Preserve a session streaming listener manager
across property access)
> Avoid listener callback deadlocks and preserve retries during streaming bus
> shutdown
> ------------------------------------------------------------------------------------
>
> Key: SPARK-61097
> URL: https://issues.apache.org/jira/browse/SPARK-61097
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> The session-shared manager and synchronous query-start callbacks landed in
> SPARK-61162 (PR #353, commite8b1d42). This ticket now tracks the remaining
> shutdown and failure behavior of that shared bus.
> Current upstream holds an operation mutex while joining the listener worker.
> If an external close or last-listener removal waits for that worker while its
> callback calls a manager method, the two threads can deadlock. A failed
> manager close also clears registrations and discards a live worker handle,
> preventing removal retry. Quiet workers and progress trackers can retain
> native session/bus state without local cancellation; stop after a failed
> server release can leave the stream waiting.
> The follow-up must avoid holding lifecycle locks across RPCs or joins,
> preserve ordinary failed-removal/close retry state, and cancel local listener
> work during terminal session shutdown even if server cleanup fails. It must
> retain the newly added listener.spark, manager._sqlb, listener ordering,
> native listener introspection, and synchronous start behavior.
> Deterministic callback/RPC gates, failure retry, stop cleanup, and ownership
> tests are being prepared. Source traces establish the issue; native
> validation and independent review are pending.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
