[
https://issues.apache.org/jira/browse/SPARK-60131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitthal updated SPARK-60131:
----------------------------
Description:
Problem
In the Scala Spark Connect client, removing a streaming listener that was never
registered can stop event delivery for another registered listener. The removal
code checks whether one listener remains, but does not check whether it is
removing that listener before stopping the shared bus.
How to reproduce
1. Register listener A with spark.streams.addListener(A).
2. Call spark.streams.removeListener(B), where B is a different listener that
was never registered.
3. The client sends a command to remove the shared server-side listener bus and
interrupts its event worker.
Expected behavior
Removing B should do nothing. Listener A should remain registered and continue
receiving events. This also matches the membership check already used by Python
Connect.
Why this fix is needed
Removing an unrelated listener should not disable event handling for a
registered listener. The server cannot prevent this mistake because the removal
command addresses the shared bus, rather than an individual client listener.
Proposed change and testing
Check that the listener is registered before stopping the shared bus. Keep the
existing behavior for removing a registered listener, including the last
listener.
A focused test against the production removal method fails before the fix
because it records an unwanted bus removal command. After the fix, that test,
the registered-listener removal test, and SparkSessionSuite pass: 12 tests in
total. Module style checks also pass. The existing live-server streaming
listener test also fails before the fix because the termination event is not
delivered, and passes after the fix. It checks progress and termination events
for both listener interface variants. No tests were ignored in either live run.
Related issues
SPARK-59847 and SPARK-59623 concern Python Connect deadlocks when removing the
last registered listener. This issue concerns removing an unregistered listener
in the Scala client, so the trigger and fix are different.
SPARK-58000 concerns Scala listener-test flakiness caused by interrupted worker
cleanup. It is related lifecycle context, but does not address this missing
membership check.
Affected versions
The same removal path is present in the Spark 4.0.0 source. The regression was
run on the current 5.0.0-SNAPSHOT master checkout; no released-version runtime
test was run.
was:
Problem
In the Scala Spark Connect client, removing a streaming listener that was never
registered can stop event delivery for another registered listener. The removal
code checks whether one listener remains, but does not check whether it is
removing that listener before stopping the shared bus.
How to reproduce
1. Register listener A with spark.streams.addListener(A).
2. Call spark.streams.removeListener(B), where B is a different listener that
was never registered.
3. The client sends a command to remove the shared server-side listener bus and
interrupts its event worker.
Expected behavior
Removing B should do nothing. Listener A should remain registered and continue
receiving events. This also matches the membership check already used by Python
Connect.
Why this fix is needed
Removing an unrelated listener should not disable event handling for a
registered listener. The server cannot prevent this mistake because the removal
command addresses the shared bus, rather than an individual client listener.
Proposed change and testing
Check that the listener is registered before stopping the shared bus. Keep the
existing behavior for removing a registered listener, including the last
listener.
A focused test against the production removal method fails before the fix
because it records an unwanted bus removal command. After the fix, that test,
the registered-listener removal test, and SparkSessionSuite pass: 12 tests in
total. Module style checks also pass. The tests use a controlled worker and
command recorder; no live Spark server test was run.
Related issues
SPARK-59847 and SPARK-59623 concern Python Connect deadlocks when removing the
last registered listener. This issue concerns removing an unregistered listener
in the Scala client, so the trigger and fix are different.
SPARK-58000 concerns Scala listener-test flakiness caused by interrupted worker
cleanup. It is related lifecycle context, but does not address this missing
membership check.
Affected versions
The same removal path is present in the Spark 4.0.0 source. The regression was
run on the current 5.0.0-SNAPSHOT master checkout; no released-version runtime
test was run.
> Removing an unregistered Scala Connect streaming listener stops the shared bus
> ------------------------------------------------------------------------------
>
> Key: SPARK-60131
> URL: https://issues.apache.org/jira/browse/SPARK-60131
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 4.0.0, 5.0.0
> Reporter: Vitthal
> Priority: Minor
>
> Problem
> In the Scala Spark Connect client, removing a streaming listener that was
> never registered can stop event delivery for another registered listener. The
> removal code checks whether one listener remains, but does not check whether
> it is removing that listener before stopping the shared bus.
> How to reproduce
> 1. Register listener A with spark.streams.addListener(A).
> 2. Call spark.streams.removeListener(B), where B is a different listener that
> was never registered.
> 3. The client sends a command to remove the shared server-side listener bus
> and interrupts its event worker.
> Expected behavior
> Removing B should do nothing. Listener A should remain registered and
> continue receiving events. This also matches the membership check already
> used by Python Connect.
> Why this fix is needed
> Removing an unrelated listener should not disable event handling for a
> registered listener. The server cannot prevent this mistake because the
> removal command addresses the shared bus, rather than an individual client
> listener.
> Proposed change and testing
> Check that the listener is registered before stopping the shared bus. Keep
> the existing behavior for removing a registered listener, including the last
> listener.
> A focused test against the production removal method fails before the fix
> because it records an unwanted bus removal command. After the fix, that test,
> the registered-listener removal test, and SparkSessionSuite pass: 12 tests in
> total. Module style checks also pass. The existing live-server streaming
> listener test also fails before the fix because the termination event is not
> delivered, and passes after the fix. It checks progress and termination
> events for both listener interface variants. No tests were ignored in either
> live run.
> Related issues
> SPARK-59847 and SPARK-59623 concern Python Connect deadlocks when removing
> the last registered listener. This issue concerns removing an unregistered
> listener in the Scala client, so the trigger and fix are different.
> SPARK-58000 concerns Scala listener-test flakiness caused by interrupted
> worker cleanup. It is related lifecycle context, but does not address this
> missing membership check.
> Affected versions
> The same removal path is present in the Spark 4.0.0 source. The regression
> was run on the current 5.0.0-SNAPSHOT master checkout; no released-version
> runtime test was run.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
