This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new e800766f2f [Improve][Zeta] Exclude engine_checkpoint_monitor from IMap
persistence (#13015)
e800766f2f is described below
commit e800766f2fdc394824c055e962d1ee362d881531
Author: Doyeon Kim <[email protected]>
AuthorDate: Tue Nov 10 15:09:22 2026 +0000
[Improve][Zeta] Exclude engine_checkpoint_monitor from IMap persistence
(#13015)
---
docs/en/engines/zeta/hybrid-cluster-deployment.md | 10 ++--
.../engines/zeta/separated-cluster-deployment.md | 10 ++--
.../en/getting-started/kubernetes/configuration.md | 12 +----
docs/zh/engines/zeta/hybrid-cluster-deployment.md | 9 ++--
.../engines/zeta/separated-cluster-deployment.md | 9 ++--
.../zh/getting-started/kubernetes/configuration.md | 12 +----
.../engine/e2e/ClusterFaultToleranceIT.java | 18 +++++++
.../engine/server/persistence/FileMapStore.java | 21 ++++++--
.../server/persistence/FileMapStoreTest.java | 63 +++++++++++++++++++---
9 files changed, 118 insertions(+), 46 deletions(-)
diff --git a/docs/en/engines/zeta/hybrid-cluster-deployment.md
b/docs/en/engines/zeta/hybrid-cluster-deployment.md
index 87a3deb7c3..959087969f 100644
--- a/docs/en/engines/zeta/hybrid-cluster-deployment.md
+++ b/docs/en/engines/zeta/hybrid-cluster-deployment.md
@@ -338,10 +338,12 @@ map:
fs.defaultFS: file:///
```
-Note: `engine_runningJobMetrics` stores high-frequency runtime metrics
snapshots and is
-intentionally excluded from persistent IMAP storage even when `map.engine*`
uses `map-store`. This
-avoids excessive WAL growth for observability-only state. After an engine
restart, running-job
-metrics are rebuilt from subsequent reports instead of continuing from the
pre-restart snapshot.
+Note: `engine_runningJobMetrics` stores high-frequency runtime metrics
snapshots and
+`engine_checkpoint_monitor` stores the checkpoint overview and history served
by the REST API.
+Both are intentionally excluded from persistent IMAP storage even when
`map.engine*` uses
+`map-store`. This avoids excessive WAL growth for observability-only state.
After an engine
+restart, running-job metrics are rebuilt from subsequent reports and the
checkpoint overview and
+history start again from the next checkpoint, instead of continuing from the
pre-restart state.
If using OSS, you can configure it as follows:
diff --git a/docs/en/engines/zeta/separated-cluster-deployment.md
b/docs/en/engines/zeta/separated-cluster-deployment.md
index bcc20d1393..f15dc52635 100644
--- a/docs/en/engines/zeta/separated-cluster-deployment.md
+++ b/docs/en/engines/zeta/separated-cluster-deployment.md
@@ -290,10 +290,12 @@ map:
fs.defaultFS: file:///
```
-Note: `engine_runningJobMetrics` stores high-frequency runtime metrics
snapshots and is
-intentionally excluded from persistent IMAP storage even when `map.engine*`
uses `map-store`. This
-avoids excessive WAL growth for observability-only state. After an engine
restart, running-job
-metrics are rebuilt from subsequent reports instead of continuing from the
pre-restart snapshot.
+Note: `engine_runningJobMetrics` stores high-frequency runtime metrics
snapshots and
+`engine_checkpoint_monitor` stores the checkpoint overview and history served
by the REST API.
+Both are intentionally excluded from persistent IMAP storage even when
`map.engine*` uses
+`map-store`. This avoids excessive WAL growth for observability-only state.
After an engine
+restart, running-job metrics are rebuilt from subsequent reports and the
checkpoint overview and
+history start again from the next checkpoint, instead of continuing from the
pre-restart state.
If you use OSS, you can configure it like this:
diff --git a/docs/en/getting-started/kubernetes/configuration.md
b/docs/en/getting-started/kubernetes/configuration.md
index 94a9888866..b94aeefaf4 100644
--- a/docs/en/getting-started/kubernetes/configuration.md
+++ b/docs/en/getting-started/kubernetes/configuration.md
@@ -180,20 +180,12 @@ MapStore uses `FileMapStoreFactory`. The `type: hdfs`
entry is the identifier of
### Checkpoint Monitor MapStore
-If you do not want to persist checkpoint monitor data, add an explicit
override for `engine_checkpoint_monitor`:
-
-```yaml
-hazelcast:
- map:
- engine_checkpoint_monitor:
- map-store:
- enabled: false
-```
+`engine_checkpoint_monitor` is never written to the MapStore, even when
`engine*` enables it, so it needs no override.
:::info Note
This IMap stores checkpoint overview and history data used by REST APIs and UI
observability. It is not the authoritative checkpoint recovery state.
Checkpoint recovery state is written by `seatunnel.engine.checkpoint.storage`
to HDFS, S3, OSS, or another configured checkpoint storage backend.
-This override is optional. Use it when you want to avoid persisting
observability-only data and reduce MapStore/WAL write amplification. Omit it if
you want checkpoint monitor overview/history to use the same MapStore settings
as other `engine*` IMaps.
+Like `engine_runningJobMetrics`, it is excluded from MapStore persistence to
avoid MapStore/WAL write amplification. After every member restarts, the
checkpoint overview and history start again from the next checkpoint. An
existing `engine_checkpoint_monitor` override with `enabled: false` is no
longer needed and can be removed.
:::
### HDFS
diff --git a/docs/zh/engines/zeta/hybrid-cluster-deployment.md
b/docs/zh/engines/zeta/hybrid-cluster-deployment.md
index b59101405f..58de25327d 100644
--- a/docs/zh/engines/zeta/hybrid-cluster-deployment.md
+++ b/docs/zh/engines/zeta/hybrid-cluster-deployment.md
@@ -334,10 +334,11 @@ map:
fs.defaultFS: file:///
```
-说明:`engine_runningJobMetrics` 保存的是高频运行时指标快照。即使通过 `map.engine*`
-配置了 `map-store`,它也会被有意排除在持久化 IMAP 存储之外,以避免仅用于可观测性的状态导致
-WAL 持续膨胀。Engine 重启后,running-job metrics 不会延续重启前的 snapshot,而是由后续
-report 重新构建。
+说明:`engine_runningJobMetrics` 保存的是高频运行时指标快照,`engine_checkpoint_monitor`
+保存的是 REST API 使用的 checkpoint overview/history。即使通过 `map.engine*` 配置了
+`map-store`,它们也会被有意排除在持久化 IMAP 存储之外,以避免仅用于可观测性的状态导致 WAL
+持续膨胀。Engine 重启后,running-job metrics 由后续 report 重新构建,checkpoint
+overview/history 从下一次 checkpoint 开始重新记录,而不会延续重启前的状态。
如果您使用 OSS,可以像这样配置:
diff --git a/docs/zh/engines/zeta/separated-cluster-deployment.md
b/docs/zh/engines/zeta/separated-cluster-deployment.md
index 311b0d4f01..e78b5f9645 100644
--- a/docs/zh/engines/zeta/separated-cluster-deployment.md
+++ b/docs/zh/engines/zeta/separated-cluster-deployment.md
@@ -295,10 +295,11 @@ map:
fs.defaultFS: file:///
```
-说明:`engine_runningJobMetrics` 保存的是高频运行时指标快照。即使通过 `map.engine*`
-配置了 `map-store`,它也会被有意排除在持久化 IMAP 存储之外,以避免仅用于可观测性的状态导致
-WAL 持续膨胀。Engine 重启后,running-job metrics 不会延续重启前的 snapshot,而是由后续
-report 重新构建。
+说明:`engine_runningJobMetrics` 保存的是高频运行时指标快照,`engine_checkpoint_monitor`
+保存的是 REST API 使用的 checkpoint overview/history。即使通过 `map.engine*` 配置了
+`map-store`,它们也会被有意排除在持久化 IMAP 存储之外,以避免仅用于可观测性的状态导致 WAL
+持续膨胀。Engine 重启后,running-job metrics 由后续 report 重新构建,checkpoint
+overview/history 从下一次 checkpoint 开始重新记录,而不会延续重启前的状态。
如果您使用 OSS,可以像这样配置:
diff --git a/docs/zh/getting-started/kubernetes/configuration.md
b/docs/zh/getting-started/kubernetes/configuration.md
index 35a1e210df..2f961a22c9 100644
--- a/docs/zh/getting-started/kubernetes/configuration.md
+++ b/docs/zh/getting-started/kubernetes/configuration.md
@@ -180,20 +180,12 @@ MapStore 使用 `FileMapStoreFactory`。其中 `type: hdfs` 是 IMap
文件存
### Checkpoint Monitor MapStore
-如果不希望持久化 checkpoint monitor 数据,可以为 `engine_checkpoint_monitor` 添加显式覆盖:
-
-```yaml
-hazelcast:
- map:
- engine_checkpoint_monitor:
- map-store:
- enabled: false
-```
+即使 `engine*` 启用了 MapStore,`engine_checkpoint_monitor` 也不会写入
MapStore,因此无需为它添加覆盖。
:::info 说明
这个 IMap 存储的是 REST API 和 UI 观测使用的 checkpoint overview/history 数据,并不是用于恢复的权威
checkpoint 状态。真正的 checkpoint 恢复状态由 `seatunnel.engine.checkpoint.storage` 写入
HDFS、S3、OSS 或其他已配置的 checkpoint 存储后端。
-这个覆盖是可选的。需要避免持久化仅用于观测的数据、减少 MapStore/WAL 写放大时可以使用;如果希望 checkpoint monitor 的
overview/history 与其他 `engine*` IMap 使用相同的 MapStore 配置,可以不添加该覆盖。
+与 `engine_runningJobMetrics` 一样,它被排除在 MapStore 持久化之外,以避免 MapStore/WAL
写放大。所有节点重启后,checkpoint overview/history 会从下一次 checkpoint 开始重新记录。已有的 `enabled:
false` 的 `engine_checkpoint_monitor` 覆盖不再需要,可以删除。
:::
### HDFS
diff --git
a/seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/java/org/apache/seatunnel/engine/e2e/ClusterFaultToleranceIT.java
b/seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/java/org/apache/seatunnel/engine/e2e/ClusterFaultToleranceIT.java
index f99cca0950..7f352753ce 100644
---
a/seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/java/org/apache/seatunnel/engine/e2e/ClusterFaultToleranceIT.java
+++
b/seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/java/org/apache/seatunnel/engine/e2e/ClusterFaultToleranceIT.java
@@ -31,6 +31,7 @@ import org.apache.seatunnel.engine.common.config.JobConfig;
import org.apache.seatunnel.engine.common.config.SeaTunnelConfig;
import org.apache.seatunnel.engine.common.job.JobStatus;
import org.apache.seatunnel.engine.server.SeaTunnelServerStarter;
+import
org.apache.seatunnel.engine.server.common.statestore.EngineStateStoreNames;
import org.awaitility.Awaitility;
import org.junit.jupiter.api.Assertions;
@@ -42,6 +43,7 @@ import com.hazelcast.client.config.ClientConfig;
import com.hazelcast.config.Config;
import com.hazelcast.core.EntryView;
import com.hazelcast.instance.impl.HazelcastInstanceImpl;
+import com.hazelcast.map.IMap;
import lombok.NonNull;
import lombok.extern.slf4j.Slf4j;
@@ -849,6 +851,22 @@ public class ClusterFaultToleranceIT {
Assertions.assertTrue(lineNumberFromDir > 1);
});
+ // engine_checkpoint_monitor matches the engine* map-store config,
but it only holds
+ // observability state: it keeps the job's checkpoint overview in
memory and writes
+ // nothing to the IMap storage, next to the engine maps that are
persisted.
+ IMap<Long, Object> checkpointMonitor =
+ finalNode.getMap(EngineStateStoreNames.CHECKPOINT_MONITOR);
+ Awaitility.await()
+ .atMost(60, TimeUnit.SECONDS)
+ .untilAsserted(
+ () ->
Assertions.assertTrue(checkpointMonitor.containsKey(jobId)));
+ File imapStorageRoot =
+ new File("/tmp/seatunnel/imap",
TestUtils.getClusterName(testClusterName));
+ Assertions.assertTrue(
+ new File(imapStorageRoot,
Constant.IMAP_RUNNING_JOB_STATE).isDirectory());
+ Assertions.assertFalse(
+ new File(imapStorageRoot,
EngineStateStoreNames.CHECKPOINT_MONITOR).exists());
+
Thread.sleep(5000);
// shutdown all node
node1.shutdown();
diff --git
a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/persistence/FileMapStore.java
b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/persistence/FileMapStore.java
index 16c20e9330..8592f4131d 100644
---
a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/persistence/FileMapStore.java
+++
b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/persistence/FileMapStore.java
@@ -45,6 +45,9 @@ import java.util.Properties;
* entry. Hazelcast does not delete an entry from the map store when it
expires, so the expiration
* time is stored with the entry and handed back on load; otherwise an entry
put with a TTL would be
* loaded again after a restart and never expire.
+ *
+ * <p>The auxiliary observability maps, {@link
EngineStateStoreNames#RUNNING_JOB_METRICS} and {@link
+ * EngineStateStoreNames#CHECKPOINT_MONITOR}, are never persisted, whatever
the map-store config.
*/
@Slf4j
public class FileMapStore implements EntryStore<Object, Object>,
MapLoaderLifecycleSupport {
@@ -53,11 +56,11 @@ public class FileMapStore implements EntryStore<Object,
Object>, MapLoaderLifecy
@Override
public void init(HazelcastInstance hazelcastInstance, Properties
properties, String mapName) {
- if (EngineStateStoreNames.RUNNING_JOB_METRICS.equals(mapName)) {
+ if (isAuxiliaryObservabilityMap(mapName)) {
this.mapStorage = NoOpMapStorage.INSTANCE;
log.info(
- "Skip persistence for map '{}' because runtime metrics
snapshots are auxiliary "
- + "observability state and should not write to
persistent IMAP storage.",
+ "Skip persistence for map '{}' because it holds auxiliary
observability state "
+ + "and should not write to persistent IMAP
storage.",
mapName);
return;
}
@@ -148,6 +151,18 @@ public class FileMapStore implements EntryStore<Object,
Object>, MapLoaderLifecy
return mapStorage.loadAllKeys();
}
+ /**
+ * Returns whether the map only holds observability state that nothing on
the recovery path
+ * reads. Running jobs refill these maps after a restart: workers report
runtime metrics again,
+ * and the checkpoint monitor records the next checkpoint. Persisting them
only grows the
+ * append-only storage; the checkpoint monitor alone rewrites a job's
whole overview on every
+ * checkpoint trigger, subtask acknowledgement and completion.
+ */
+ private static boolean isAuxiliaryObservabilityMap(String mapName) {
+ return EngineStateStoreNames.RUNNING_JOB_METRICS.equals(mapName)
+ || EngineStateStoreNames.CHECKPOINT_MONITOR.equals(mapName);
+ }
+
private static final class NoOpMapStorage implements IMapStorage {
private static final NoOpMapStorage INSTANCE = new NoOpMapStorage();
diff --git
a/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/persistence/FileMapStoreTest.java
b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/persistence/FileMapStoreTest.java
index d6d2491fdb..53911eced3 100644
---
a/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/persistence/FileMapStoreTest.java
+++
b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/persistence/FileMapStoreTest.java
@@ -26,6 +26,8 @@ import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.condition.EnabledOnOs;
import org.junit.jupiter.api.condition.OS;
import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
import com.hazelcast.config.Config;
import com.hazelcast.config.JoinConfig;
@@ -37,8 +39,11 @@ import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.map.EntryLoader.MetadataAwareValue;
import com.hazelcast.map.IMap;
+import java.nio.file.Files;
import java.nio.file.Path;
+import java.util.Arrays;
import java.util.Collections;
+import java.util.List;
import java.util.Properties;
import java.util.concurrent.TimeUnit;
@@ -114,18 +119,19 @@ public class FileMapStoreTest {
"Exception message should identify the unknown type. Got: " +
msg);
}
- @Test
- public void testMetricsSnapshotMapSkipsPersistentStorageInitialization() {
+ @ParameterizedTest
+ @ValueSource(
+ strings = {
+ EngineStateStoreNames.RUNNING_JOB_METRICS,
+ EngineStateStoreNames.CHECKPOINT_MONITOR
+ })
+ public void
testAuxiliaryObservabilityMapSkipsPersistentStorageInitialization(String
mapName) {
FileMapStore store = new FileMapStore();
Properties props = new Properties();
props.setProperty("type", "non-existent-storage-type");
Assertions.assertDoesNotThrow(
- () ->
- store.init(
- mock(HazelcastInstance.class),
- props,
- EngineStateStoreNames.RUNNING_JOB_METRICS));
+ () -> store.init(mock(HazelcastInstance.class), props,
mapName));
Assertions.assertEquals(Collections.emptySet(), store.loadAllKeys());
Assertions.assertDoesNotThrow(() -> store.store("k", new
MetadataAwareValue<>("v")));
Assertions.assertDoesNotThrow(() -> store.delete("k"));
@@ -133,6 +139,49 @@ public class FileMapStoreTest {
store.destroy();
}
+ /**
+ * The auxiliary observability maps match the {@code engine*} map-store
config like every other
+ * engine map. Verifies that they are not written to the IMap storage and
so start empty after a
+ * restart, while an engine state map next to them is still persisted and
loaded back.
+ */
+ @Test
+ public void testAuxiliaryObservabilityMapsAreNotPersistedAcrossRestart() {
+ String clusterName = "file-map-store-auxiliary-" + System.nanoTime();
+ String persistedMapName = Constant.IMAP_RUNNING_JOB_STATE;
+ List<String> auxiliaryMapNames =
+ Arrays.asList(
+ EngineStateStoreNames.RUNNING_JOB_METRICS,
+ EngineStateStoreNames.CHECKPOINT_MONITOR);
+
+ HazelcastInstance writer =
Hazelcast.newHazelcastInstance(hazelcastConfig(clusterName));
+ try {
+ writer.getMap(persistedMapName).put(1L, "persisted");
+ auxiliaryMapNames.forEach(mapName ->
writer.getMap(mapName).put(1L, "auxiliary"));
+ } finally {
+ writer.shutdown();
+ }
+
+ Path storageRoot = tempDir.resolve("test-cluster");
+
Assertions.assertTrue(Files.isDirectory(storageRoot.resolve(persistedMapName)));
+ auxiliaryMapNames.forEach(
+ mapName ->
+ Assertions.assertFalse(
+ Files.exists(storageRoot.resolve(mapName)),
+ mapName + " must not be written to the IMap
storage"));
+
+ HazelcastInstance restarted =
Hazelcast.newHazelcastInstance(hazelcastConfig(clusterName));
+ try {
+ Assertions.assertEquals("persisted",
restarted.getMap(persistedMapName).get(1L));
+ auxiliaryMapNames.forEach(
+ mapName ->
+ Assertions.assertTrue(
+ restarted.getMap(mapName).isEmpty(),
+ mapName + " must not be loaded again after
a restart"));
+ } finally {
+ restarted.shutdown();
+ }
+ }
+
/**
* Hazelcast does not delete an entry from the map store when its TTL
expires. Verifies that
* after a restart an expired entry is not loaded again, and an entry put
with a TTL keeps its