voonhous opened a new issue, #19068:
URL: https://github.com/apache/hudi/issues/19068
## Bug Description
**What happened:**
`TestHoodieClientMultiWriter.testHoodieClientBasicMultiWriterWithEarlyConflictDetectionDirect`
started failing in CI at the start of this week. It passed reliably before. In
the surefire summary it surfaces as:
```
Unable to instantiate class
org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider
```
but that ZK message is a downstream symptom. The real regression is a
heartbeat-expiry timing change.
**Root cause -- a regression from #18904 (`c6f38c6b6e84`, merged
2026-06-22):**
`fix: prevent heartbeat timer from being permanently killed by slow or
delayed heartbeats` raised the default
`hoodie.client.heartbeat.tolerable.misses` from 2 to 10:
```diff
public static final ConfigProperty<Integer>
CLIENT_HEARTBEAT_NUM_TOLERABLE_MISSES = ConfigProperty
.key("hoodie.client.heartbeat.tolerable.misses")
- .defaultValue(2)
+ .defaultValue(10)
```
The heartbeat expiry window is `interval x tolerableMisses`
(`HoodieHeartbeatClient.java:86`, enforced at `:228`):
```java
this.maxAllowableHeartbeatIntervalInMs = this.heartbeatIntervalInMs *
numTolerableHeartbeatMisses;
...
if (currentTime - lastHeartbeatTime >
this.maxAllowableHeartbeatIntervalInMs) { return true; /* expired */ }
```
In this test, commit `004` runs with `config4`, whose heartbeat interval is
`heartBeatIntervalForCommit4 = 3000ms`
(`TestHoodieClientMultiWriter.java:462,516`), and which does **not** pin
tolerable-misses, so it inherits the new default. The test writes commit
`003`'s heartbeat file, then waits, expecting `003` to expire so commit `004`
does not see it as a live conflict:
```java
// TestHoodieClientMultiWriter.java:523-525
// Wait for heart beat expired for failed commitTime3 "003"
// Otherwise commit4 still can see conflict between failed write 003.
Thread.sleep(heartBeatIntervalForCommit4 * 2); // = 6000ms
```
| | expiry window (`3000 x misses`) | test sleep | result |
|---|---|---|---|
| before #18904 (misses = 2) | 6000 ms | 6000 ms | borderline; `003` just
expires, commit `004` succeeds |
| after #18904 (misses = 10) | 30000 ms | 6000 ms | `003` still "alive";
commit `004` hits early conflict |
`sleep(interval * 2)` was written assuming `misses = 2`. With the new
default the 6s wait is 5x too short, so commit `004`'s `assertDoesNotThrow`
(line 528) fails -- the early-conflict detector fires on `004`. This is the CI
"Run 1" failure
(`...testHoodieClientBasicMultiWriterWithEarlyConflictDetection:528`).
(The sibling test path at line 1085 explicitly pins
`withHeartbeatTolerableMisses(1)`, so it is immune. Only this early-conflict
test, which relies on the default, regressed.)
**Why it then surfaces as the ZooKeeper error (pre-existing, latent
amplifiers):**
The Run 1 failure throws before the `TestingServer` is closed -- cleanup is
not in a `finally`:
```java
// TestHoodieClientMultiWriter.java:543-546 (end of method body)
FileIOUtils.deleteDirectory(new File(basePath));
if (server != null) {
server.close();
}
```
With surefire `forkCount=1, reuseForks=true` (`pom.xml:2141`), the shard
runs in one long-lived JVM, so the un-closed Curator `TestingServer` leaks for
the rest of the JVM. On ZooKeeper 3.5.7 (`pom.xml:238`) `TestingServer` starts
an embedded Jetty `AdminServer` on the fixed default port 8080 (Curator
randomizes only the ZK client port). Every later rerun then constructs a new
`TestingServer` whose admin server fails with `BindException: Address already
in use` on 8080, and `LockManager.getLockProvider` ->
`ReflectionUtils.loadClass` rethrows it as the generic `Unable to instantiate
class ... ZookeeperBasedLockProvider`. That is the Run 2/3/4 cascade below.
**What you expected:**
The test should remain valid under the new `tolerable.misses` default, and a
single in-test failure should not be able to leak the embedded ZooKeeper and
poison every subsequent rerun in a reused JVM.
**Steps to reproduce:**
1. On master at or after #18904 (`c6f38c6b6e84`, 2026-06-22), run the
`test-spark-java17-java-tests-part2` Spark suite (`forkCount=1,
reuseForks=true`).
2. `testHoodieClientBasicMultiWriterWithEarlyConflictDetectionDirect` Run 1
fails at line 528 because commit `004`'s heartbeat-expiry wait (6s) is now
shorter than the expiry window (`10 x 3000 = 30s`), so the early-conflict
detector fires.
3. The failure leaves the `TestingServer` un-closed; Runs 2-4 then fail with
`Unable to instantiate class ... ZookeeperBasedLockProvider` (root cause:
`BindException` on port 8080).
## Environment
**Hudi version:** master at/after #18904 (`c6f38c6b6e84`, 2026-06-22);
reproduced on the `spark3.5` / `spark4.x` java17 part2 CI shards
**Query engine:** Spark (Spark 3.5 / 4.0 / 4.1 / 4.2 part2 java-tests shards)
**Relevant configs:** `hoodie.client.heartbeat.tolerable.misses` default 2
-> 10 (#18904); test commit-004 heartbeat interval = 3000 ms; ZooKeeper 3.5.7
(`pom.xml:238`); surefire `forkCount=1, reuseForks=true` (`pom.xml:2141`)
## Logs and Stack Trace
Source: GitHub Actions job `test-spark-java17-java-tests-part2 (scala-2.13,
spark3.5, hudi-spark-datasource_hudi-spark3.5.x)`.
### Run 1 -- the real regression: early conflict fires on commit 004
(assertDoesNotThrow at line 528)
```
Caused by: org.apache.hudi.exception.HoodieEarlyConflictDetectionException:
java.util.ConcurrentModificationException: Early conflict detected but cannot
resolve conflicts for overlapping writes
at
org.apache.hudi.table.marker.SimpleDirectMarkerBasedDetectionStrategy.resolveMarkerConflict(SimpleDirectMarkerBasedDetectionStrategy.java:74)
at
org.apache.hudi.table.marker.SimpleDirectMarkerBasedDetectionStrategy.detectAndResolveConflictIfNecessary(SimpleDirectMarkerBasedDetectionStrategy.java:80)
at
org.apache.hudi.table.marker.SimpleTransactionDirectMarkerBasedDetectionStrategy.detectAndResolveConflictIfNecessary(SimpleTransactionDirectMarkerBasedDetectionStrategy.java:54)
at
org.apache.hudi.table.marker.DirectWriteMarkers.createWithEarlyConflictDetection(DirectWriteMarkers.java:183)
at
org.apache.hudi.table.marker.WriteMarkers.create(WriteMarkers.java:91)
at
org.apache.hudi.io.HoodieWriteHandle.createMarkerFile(HoodieWriteHandle.java:206)
at
org.apache.hudi.io.HoodieWriteMergeHandle.initMarkerFileAndFileWriter(HoodieWriteMergeHandle.java:179)
at
org.apache.hudi.io.HoodieWriteMergeHandle.<init>(HoodieWriteMergeHandle.java:128)
at
org.apache.hudi.io.FileGroupReaderBasedMergeHandle.<init>(FileGroupReaderBasedMergeHandle.java:133)
at
org.apache.hudi.io.FileGroupReaderBasedMergeHandle.<init>(FileGroupReaderBasedMergeHandle.java:113)
... 40 more
Caused by: java.util.ConcurrentModificationException: Early conflict
detected but cannot resolve conflicts for overlapping writes
... 50 more
Driver stacktrace:
at
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:152)
at
org.junit.jupiter.api.AssertDoesNotThrow.createAssertionFailedError(AssertDoesNotThrow.java:84)
at
org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:53)
at
org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:36)
at
org.junit.jupiter.api.Assertions.assertDoesNotThrow(Assertions.java:3199)
at
org.apache.hudi.client.TestHoodieClientMultiWriter.testHoodieClientBasicMultiWriterWithEarlyConflictDetection(TestHoodieClientMultiWriter.java:528)
at
org.apache.hudi.client.TestHoodieClientMultiWriter.testHoodieClientBasicMultiWriterWithEarlyConflictDetectionDirect(TestHoodieClientMultiWriter.java:428)
```
### Surefire rerun summary -- Run 1 leaks the server; Runs 2-4 then hit the
8080 collision
```
[ERROR] Run 1:
TestHoodieClientMultiWriter.testHoodieClientBasicMultiWriterWithEarlyConflictDetectionDirect:428->testHoodieClientBasicMultiWriterWithEarlyConflictDetection:528
Unexpected exception thrown: org.apache.spark.SparkException: Job aborted due
to stage failure: Task 1 in stage 92.0 failed 1 times, most recent failure:
Lost task 1.0 in stage 92.0 (TID 198) (...executor driver):
org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType
UPDATE for partition :1
[ERROR] Run 2:
TestHoodieClientMultiWriter.testHoodieClientBasicMultiWriterWithEarlyConflictDetectionDirect:428->testHoodieClientBasicMultiWriterWithEarlyConflictDetection:490->createCommitWithInserts:1686->createCommitWithInserts:1693->HoodieClientTestBase.insertFirstBatch:270->HoodieClientTestBase.writeBatch:488->HoodieClientTestBase.writeBatchHelper:528
» Hoodie Unable to instantiate class
org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider
[ERROR] Run 3:
TestHoodieClientMultiWriter.testHoodieClientBasicMultiWriterWithEarlyConflictDetectionDirect:428->testHoodieClientBasicMultiWriterWithEarlyConflictDetection:490->createCommitWithInserts:1686->createCommitWithInserts:1693->HoodieClientTestBase.insertFirstBatch:270->HoodieClientTestBase.writeBatch:488->HoodieClientTestBase.writeBatchHelper:528
» Hoodie Unable to instantiate class
org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider
[ERROR] Run 4:
TestHoodieClientMultiWriter.testHoodieClientBasicMultiWriterWithEarlyConflictDetectionDirect:428->testHoodieClientBasicMultiWriterWithEarlyConflictDetection:490->createCommitWithInserts:1686->createCommitWithInserts:1693->HoodieClientTestBase.insertFirstBatch:270->HoodieClientTestBase.writeBatch:488->HoodieClientTestBase.writeBatchHelper:528
» Hoodie Unable to instantiate class
org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider
```
### Runs 2-4 root cause: ZooKeeper AdminServer cannot bind port 8080 (leaked
TestingServer)
```
2095057 [Thread-206236] ERROR org.apache.curator.test.TestingZooKeeperServer
[] - From testing server (random state: false) for instance:
InstanceSpec{dataDirectory=/tmp/1782375987563-0, port=43849,
electionPort=40245, quorumPort=40159, deleteDataDirectoryOnClose=true,
serverId=2, tickTime=-1, maxClientCnxns=-1}
org.apache.curator.test.InstanceSpec@ab49
org.apache.zookeeper.server.admin.AdminServer$AdminServerException: Problem
starting AdminServer on address 0.0.0.0, port 8080 and command URL /commands
at
org.apache.zookeeper.server.admin.JettyAdminServer.start(JettyAdminServer.java:189)
at
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:153)
at
org.apache.curator.test.TestingZooKeeperMain.runFromConfig(TestingZooKeeperMain.java:73)
at
org.apache.curator.test.TestingZooKeeperServer$1.run(TestingZooKeeperServer.java:148)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.io.IOException: Failed to bind to /0.0.0.0:8080
at
org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
at
org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
at
org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at
org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.server.Server.doStart(Server.java:401)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.apache.zookeeper.server.admin.JettyAdminServer.start(JettyAdminServer.java:180)
... 4 more
Caused by: java.net.BindException: Address already in use
at java.base/sun.nio.ch.Net.bind0(Native Method)
at java.base/sun.nio.ch.Net.bind(Net.java:567)
at
java.base/sun.nio.ch.ServerSocketChannelImpl.netBind(ServerSocketChannelImpl.java:337)
```
```
[ERROR]
org.apache.hudi.client.TestHoodieClientMultiWriter.testHoodieClientBasicMultiWriterWithEarlyConflictDetectionDirect(String,
String)[3] -- Time elapsed: 13.25 s <<< ERROR!
org.apache.hudi.exception.HoodieException: Unable to instantiate class
org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider
at
org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:73)
at
org.apache.hudi.client.transaction.lock.LockManager.getLockProvider(LockManager.java:122)
at
org.apache.hudi.client.transaction.lock.LockManager.lambda$lock$20c251e3$1(LockManager.java:81)
at org.apache.hudi.common.util.RetryHelper.start(RetryHelper.java:94)
at
org.apache.hudi.client.transaction.lock.LockManager.lock(LockManager.java:78)
at
org.apache.hudi.client.transaction.TransactionManager.beginStateChange(TransactionManager.java:61)
at
org.apache.hudi.client.BaseHoodieWriteClient.startCommit(BaseHoodieWriteClient.java:1131)
at
org.apache.hudi.client.WriteClientTestUtils.startCommitWithTime(WriteClientTestUtils.java:42)
at
org.apache.hudi.testutils.HoodieClientTestBase.writeBatchHelper(HoodieClientTestBase.java:528)
at
org.apache.hudi.testutils.HoodieClientTestBase.writeBatch(HoodieClientTestBase.java:488)
at
org.apache.hudi.testutils.HoodieClientTestBase.insertFirstBatch(HoodieClientTestBase.java:270)
at
org.apache.hudi.client.TestHoodieClientMultiWriter.createCommitWithInserts(TestHoodieClientMultiWriter.java:1693)
at
org.apache.hudi.client.TestHoodieClientMultiWriter.createCommitWithInserts(TestHoodieClientMultiWriter.java:1686)
at
org.apache.hudi.client.TestHoodieClientMultiWriter.testHoodieClientBasicMultiWriterWithEarlyConflictDetection(TestHoodieClientMultiWriter.java:490)
at
org.apache.hudi.client.TestHoodieClientMultiWriter.testHoodieClientBasicMultiWriterWithEarlyConflictDetectionDirect(TestHoodieClientMultiWriter.java:428)
```
## Proposed fix
Primary (addresses the regression):
- In `buildWriteConfigForEarlyConflictDetect` (the early-conflict test
config), pin `withHeartbeatTolerableMisses(...)` so the test's heartbeat-expiry
wait is independent of the global default, or scale the wait to
`(tolerableMisses + 1) x interval`. Pinning is preferred -- deterministic and
fast.
Secondary (latent test-infra hardening, so a single failure cannot cascade):
- Wrap the `TestingServer` lifecycle in `try/finally` so it is always closed
even when an assertion fails.
- Disable the ZooKeeper admin server in tests (it is unused) by setting
`zookeeper.admin.enableServer=false` before constructing `TestingServer`,
removing the fixed-port-8080 collision. The same `TestingServer` pattern exists
in `TestLockManager`, `TestZookeeperBasedLockProvider`, and
`TestSimpleTransactionDirectMarkerBasedDetectionStrategyWithZKLockProvider`, so
apply it consistently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
