SEZ9 opened a new pull request, #12789:
URL: https://github.com/apache/seatunnel/pull/12789
### Purpose of this pull request
Closes #12788.
`CheckpointMonitorService` was constructed with a literal in
`SeaTunnelServer.startMaster()`:
```java
checkpointMonitorService = new CheckpointMonitorService(engineContext, 32);
```
That number is how many completed checkpoints the monitor keeps **per
pipeline** for `/jobs/checkpoints/{jobId}`
(`PipelineCheckpointOverview.addHistory` trims to it), and there was no way to
change it.
The window bounds how far back a client can compare a job against its own
past, which is what diagnosing a *gradual* slowdown needs — a job still
`RUNNING`, both row counters moving, `restoreCount` 0, and the only evidence of
change being that checkpoints take longer than they used to. How much history
32 entries buys depends entirely on the interval:
| `checkpoint.interval` | covered by 32 entries |
|---|---|
| 300000 ms (engine default) | ~2.5 hours |
| 60000 ms | ~32 minutes |
| 10000 ms (common for CDC) | ~5 minutes |
Five minutes is not enough baseline to tell a trend from noise, and a short
interval is exactly what a latency-sensitive CDC job runs with.
Adds `seatunnel.engine.checkpoint.monitor-history-size`:
```yaml
seatunnel:
engine:
checkpoint:
monitor-history-size: 128
```
-
`ServerConfigOptions.MasterServerConfigOptions.CHECKPOINT_MONITOR_HISTORY_SIZE`,
default `32`
- `CheckpointConfig.monitorHistorySize`, with an explicit setter that
rejects values below 1
- a branch in `YamlSeaTunnelDomConfigProcessor` so the key is actually read
— the checkpoint block is parsed with an explicit per-key `if/else if` chain,
so an option without a branch there is silently ignored
- `SeaTunnelServer.startMaster()` reads it instead of passing the literal
The history is held in memory per pipeline, so a larger window costs memory
proportional to the number of running pipelines. That is documented rather than
left implicit.
### Does this PR introduce _any_ user-facing change?
A new optional configuration key, and two documentation sections (en + zh)
describing it and the interval/window relationship above.
No behaviour change by default: the option defaults to 32, which is the
value that was hardcoded. A cluster that does not set it behaves exactly as
before.
### How was this patch tested or is it not required?
`YamlSeaTunnelConfigParserTest` is extended: the test fixture
`seatunnel.yaml` now sets `monitor-history-size: 64` and the test asserts it is
parsed, which covers the whole path — option declaration, the YAML branch, and
the config object. Without the `YamlSeaTunnelDomConfigProcessor` branch this
test fails, since the value would fall back to the default.
Run locally against this branch:
```
[ 5 tests successful ]
[ 0 tests failed ]
```
Maven cannot resolve `build-helper-maven-plugin` in my environment, so I ran
the module's sources and this test through `javac` and the JUnit Platform
launcher directly rather than `mvn test`: `seatunnel-engine-common` compiles
with zero errors, and the five tests in the class pass. `mvn spotless:check`
passes for both touched modules. Full verification is of course down to CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
