[
https://issues.apache.org/jira/browse/SPARK-58844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohamed Gallab updated SPARK-58844:
-----------------------------------
Description:
h3. Symptom
{{EchoProtocolSuite}} intermittently fails the case {{cancel: engine sends
Cancel after Finish – accepts FinishResponse or CancelResponse}} with:
{code}
java.lang.IllegalStateException: call was half-closed
{code}
or:
{code}
Stream is already completed, no further calls are allowed
{code}
The suite also runs against our worker via {{ScalaUdfWorkerServiceSuite}},
which extends {{EchoProtocolSuite}}, so it can fail there too.
h3. Root cause
The {{EngineClient}} test helper touches its request {{StreamObserver}} from
two threads without mutual exclusion. {{sendCancel}} performs a check-then-act:
{code:language=scala}
if (requestCompleted.get()) return
requestObserver.onNext(Cancel)
{code}
Meanwhile, the gRPC response-observer callback thread, on the incoming
{{FinishResponse}}, calls {{completeRequestStream()}}:
{code:language=scala}
requestObserver.onCompleted()
{code}
If the half-close occurs between {{get()}} and {{onNext}}, {{onNext}} throws.
gRPC forbids concurrent calls on a {{StreamObserver}}. Only this test hits the
race because it is the only case that sends a trailing {{Cancel}} exactly as
the {{FinishResponse}} terminator arrives.
h3. Location
{{udf/worker/grpc/src/test/scala/org/apache/spark/udf/worker/grpc/EchoProtocolSuite.scala}}
Relevant methods: {{EngineClient.sendCancel}} and
{{EngineClient.completeRequestStream}}.
h3. Bazel targets
*
{{//udf/worker/grpc:org.apache.spark.udf.worker.grpc.EchoProtocolSuite-hive-2.3__hadoop-3.2_2.13}}
*
{{//udf/worker/grpc:org.apache.spark.udf.worker.grpc.ScalaUdfWorkerServiceSuite...}}
h3. Reproduction
The natural failure rate is very low (below approximately 0.4%); the test
passes 250+ isolated runs and typically fails only under CI scheduling load.
For a deterministic reproduction, temporarily insert {{Thread.sleep(50)}}
between {{get()}} and {{onNext}} in {{sendCancel}}. The case fails 5/5 times.
After applying the fix, it passes 5/5 times with the same widening.
h3. Impact
Intermittent red CI on the {{udf/worker/grpc}} suites.
Test-only; there is no production or runtime impact because this is an
unshipped path.
> Flaky test: `EchoProtocolSuite` "cancel: engine sends Cancel after Finish" —
> request `StreamObserver` TOCTOU race
> -----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-58844
> URL: https://issues.apache.org/jira/browse/SPARK-58844
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.3.0
> Reporter: Mohamed Gallab
> Priority: Minor
>
> h3. Symptom
> {{EchoProtocolSuite}} intermittently fails the case {{cancel: engine sends
> Cancel after Finish – accepts FinishResponse or CancelResponse}} with:
> {code}
> java.lang.IllegalStateException: call was half-closed
> {code}
> or:
> {code}
> Stream is already completed, no further calls are allowed
> {code}
> The suite also runs against our worker via {{ScalaUdfWorkerServiceSuite}},
> which extends {{EchoProtocolSuite}}, so it can fail there too.
> h3. Root cause
> The {{EngineClient}} test helper touches its request {{StreamObserver}} from
> two threads without mutual exclusion. {{sendCancel}} performs a
> check-then-act:
> {code:language=scala}
> if (requestCompleted.get()) return
> requestObserver.onNext(Cancel)
> {code}
> Meanwhile, the gRPC response-observer callback thread, on the incoming
> {{FinishResponse}}, calls {{completeRequestStream()}}:
> {code:language=scala}
> requestObserver.onCompleted()
> {code}
> If the half-close occurs between {{get()}} and {{onNext}}, {{onNext}} throws.
> gRPC forbids concurrent calls on a {{StreamObserver}}. Only this test hits
> the race because it is the only case that sends a trailing {{Cancel}} exactly
> as the {{FinishResponse}} terminator arrives.
> h3. Location
> {{udf/worker/grpc/src/test/scala/org/apache/spark/udf/worker/grpc/EchoProtocolSuite.scala}}
> Relevant methods: {{EngineClient.sendCancel}} and
> {{EngineClient.completeRequestStream}}.
> h3. Bazel targets
> *
> {{//udf/worker/grpc:org.apache.spark.udf.worker.grpc.EchoProtocolSuite-hive-2.3__hadoop-3.2_2.13}}
> *
> {{//udf/worker/grpc:org.apache.spark.udf.worker.grpc.ScalaUdfWorkerServiceSuite...}}
> h3. Reproduction
> The natural failure rate is very low (below approximately 0.4%); the test
> passes 250+ isolated runs and typically fails only under CI scheduling load.
> For a deterministic reproduction, temporarily insert {{Thread.sleep(50)}}
> between {{get()}} and {{onNext}} in {{sendCancel}}. The case fails 5/5 times.
> After applying the fix, it passes 5/5 times with the same widening.
> h3. Impact
> Intermittent red CI on the {{udf/worker/grpc}} suites.
> Test-only; there is no production or runtime impact because this is an
> unshipped path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
