Hyukjin Kwon created SPARK-57991:
------------------------------------
Summary: Deflake BarrierTaskContextSuite wall-clock skew
assertions on macOS-26
Key: SPARK-57991
URL: https://issues.apache.org/jira/browse/SPARK-57991
Project: Spark
Issue Type: Bug
Components: Spark Core, Tests
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
The `Build / Maven (Scala 2.13, JDK 21, MacOS-26)` scheduled workflow fails on
the `core` module with:
{code}
BarrierTaskContextSuite:
- successively sync with allGather and barrier *** FAILED ***
1078 was not less than or equal to 1000 (BarrierTaskContextSuite.scala:122)
{code}
Three tests in `BarrierTaskContextSuite` capture `System.currentTimeMillis()`
in each task right after a `barrier()` / `allGather()` global sync and assert
`times.max - times.min <= 1000`. That skew is bounded below by the barrier
client's own polling granularity, not by sync correctness:
`BarrierTaskContext.runBarrier` waits for the coordinator RPC in a
`Thread.sleep(1000)` loop, so two tasks can observe the release up to ~1s apart
from the poll interval alone, before any thread-scheduling or GC jitter on a
busy CI host.
SPARK-49983 previously observed 1038 and only halved the pre-barrier sleep,
which reduces arrival spread but cannot reduce the poll-granularity skew, so
the flake persists on slower runners (macOS-26 arm64 observed 1078). Reproduced
locally on macOS 26 arm64 / JDK 21: post-sync skew reached 994ms while idle.
Widen the tolerance to a documented constant that accounts for the poll
interval plus jitter, keeping the test's intent (tasks stay loosely in
lockstep; none races an extra sync ahead) without coupling to a razor-thin
wall-clock margin.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
