kalayciburak opened a new pull request, #12678:
URL: https://github.com/apache/seatunnel/pull/12678
### Purpose of this pull request
`checkpoint.storage.max-retained` is documented as the maximum number of
retained checkpoints. `CheckpointCoordinator.completePendingCheckpoint` only
deleted a batch when the completed-id queue size was a multiple of that bound
and at least twice the bound, so storage held about 2x the configured maximum
between prunes.
This drops the oldest checkpoint ids as soon as a completion would exceed
`max-retained`. The option name, default, and docs are unchanged.
Fixes #12677
### Does this PR introduce _any_ user-facing change?
Yes, compared with released Zeta behavior and current `dev`.
Previously, with `max-retained: 3`, completed checkpoints 4 and 5 stayed on
disk until checkpoint 6, so storage held up to 6 checkpoints. After this change
the queue is trimmed on the completion that crosses the bound, so at most 3
completed checkpoints remain once `completePendingCheckpoint` returns.
No config key, default, or public API is renamed. `docs/en` and `docs/zh`
already describe `max-retained` as the maximum, so they are not edited.
### How was this patch tested?
Added
`CheckpointCoordinatorTest.testRetentionPrunesAsSoonAsCompletedCheckpointsExceedMaxRetained`.
With `max-retained` 3, five completions must leave ids `[3, 4, 5]` and delete
`1` then `2`.
RED on unchanged `dev` (JAVA_HOME=8):
```
./mvnw -pl seatunnel-engine/seatunnel-engine-server test
-Dtest=CheckpointCoordinatorTest#testRetentionPrunesAsSoonAsCompletedCheckpointsExceedMaxRetained
-DfailIfNoTests=false
```
```
after checkpoint 4 retained [1, 2, 3, 4], max-retained is 3
```
GREEN, same module, full class:
```
./mvnw -pl seatunnel-engine/seatunnel-engine-server test
-Dtest=CheckpointCoordinatorTest -DfailIfNoTests=false
```
```
Tests run: 17, Failures: 0, Errors: 0, Skipped: 0
```
`CheckpointStorageRetentionPruneTest` comments and the on-disk bound were
updated to match. That integration test was not run here.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [ ] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
