[
https://issues.apache.org/jira/browse/SPARK-61097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-61097:
----------------------------
Description:
The initial shared session manager and synchronous query-start delivery landed
in SPARK-61162 (PR #353). This follow-up tracks the remaining listener
lifecycle behavior.
Terminal SparkSession.stop() and SparkConnectClient.close() must cancel local
listener registration and dispatch without waiting for a callback. Otherwise a
callback waiting on the terminal caller, a lock, an Event, or an RPC can
deadlock teardown. A registration whose event stream is already established but
not yet acknowledged must also be cancelled. Ordinary failed listener removal
must preserve registrations for retry, while generation changes prevent old
workers from clearing newer listeners.
Keep RPCs and callback joins outside the state lock. The real worker must hold
a weak bus reference so dropping the last native session/manager can release a
quiet subscription. Regression coverage includes callback reentrancy,
stop/client-close races, failed removal retry, stale generations, progress
handler lock reentrancy, and real-worker ownership release.
This follow-up preserves the original issue's connect-rust-4.3.0 release
tracking. The implementation is undergoing validation before publication.
was:
The session-shared manager and synchronous query-start callbacks landed in
SPARK-61162 (PR #353, commite8b1d42). This ticket now tracks the remaining
shutdown and failure behavior of that shared bus.
Current upstream holds an operation mutex while joining the listener worker. If
an external close or last-listener removal waits for that worker while its
callback calls a manager method, the two threads can deadlock. A failed manager
close also clears registrations and discards a live worker handle, preventing
removal retry. Quiet workers and progress trackers can retain native
session/bus state without local cancellation; stop after a failed server
release can leave the stream waiting.
The follow-up must avoid holding lifecycle locks across RPCs or joins, preserve
ordinary failed-removal/close retry state, and cancel local listener work
during terminal session shutdown even if server cleanup fails. It must retain
the newly added listener.spark, manager._sqlb, listener ordering, native
listener introspection, and synchronous start behavior.
Deterministic callback/RPC gates, failure retry, stop cleanup, and ownership
tests are being prepared. Source traces establish the issue; native validation
and independent review are pending.
Summary: Cancel pending streaming listener registration and avoid
callback deadlocks during shutdown (was: Avoid listener callback deadlocks and
preserve retries during streaming bus shutdown)
> Cancel pending streaming listener registration and avoid callback deadlocks
> during shutdown
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-61097
> URL: https://issues.apache.org/jira/browse/SPARK-61097
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> The initial shared session manager and synchronous query-start delivery
> landed in SPARK-61162 (PR #353). This follow-up tracks the remaining listener
> lifecycle behavior.
> Terminal SparkSession.stop() and SparkConnectClient.close() must cancel local
> listener registration and dispatch without waiting for a callback. Otherwise
> a callback waiting on the terminal caller, a lock, an Event, or an RPC can
> deadlock teardown. A registration whose event stream is already established
> but not yet acknowledged must also be cancelled. Ordinary failed listener
> removal must preserve registrations for retry, while generation changes
> prevent old workers from clearing newer listeners.
> Keep RPCs and callback joins outside the state lock. The real worker must
> hold a weak bus reference so dropping the last native session/manager can
> release a quiet subscription. Regression coverage includes callback
> reentrancy, stop/client-close races, failed removal retry, stale generations,
> progress handler lock reentrancy, and real-worker ownership release.
> This follow-up preserves the original issue's connect-rust-4.3.0 release
> tracking. The implementation is undergoing validation before publication.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
