goutamadwant opened a new pull request, #13027:
URL: https://github.com/apache/seatunnel/pull/13027
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code
changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB
issue](https://github.com/apache/seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the
pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix
typo in README.md doc`.
-->
### Purpose of this pull request
Fixes #12361. Related previous attempt: #12486 (closed, not merged).
This SeaTunnel change reduces retained heap during exactly-once CDC snapshot
reconciliation without changing the terminal split range or snapshot/binlog
watermark boundaries.
Previously, the fetcher retained the whole split in a `LinkedHashMap`, then
materialized another output list. A terminal split can grow beyond
`snapshot.split.size`. Exactly-once snapshot contexts also replaced the
configured Debezium queue limit with `Integer.MAX_VALUE`.
- Keep small snapshots in memory; spill after 1,024 rows or 8 MiB cumulative
encoded admission bytes to temporary ordered storage with bounded page caches.
- Preserve keyed replacement, deletion/reinsertion order, null-key behavior,
Connect values/schema metadata and record ordering.
- Decode output lazily while retaining the existing atomic LOW/rows/HIGH
emission under the checkpoint lock. Release temporary storage after emission,
failure or shutdown.
- Respect configured queue bounds in MySQL, PostgreSQL, Oracle, SQL Server,
DB2 and MongoDB contexts.
- Add regression tests, a MySQL checkpoint/restore E2E case and EN/ZH
operational documentation. No new dependency.
### Does this PR introduce _any_ user-facing change?
Yes. Large exactly-once snapshots use temporary disk space to reduce
retained heap. Existing configuration names/defaults, connector APIs, output
formats and checkpoint formats are preserved. Unfinished splits are refetched
on restore; spill files are not checkpoint state.
#### Measurements
Local paired baseline/candidate measurements used real MySQL JDBC input into
the actual fetcher, Java 11, a 256 MiB heap, two active CPUs and identical
bounded queues. Three alternating fresh-JVM pairs were run per size; values
below are medians. A fixed open-ended terminal range was reused after growing
the table. The adapter constructed the split directly and controlled
watermark/changelog delivery.
| Rows retained in split | Baseline retained heap | Candidate retained heap
| Baseline load / emit | Candidate load / emit | Candidate file bytes |
| --- | ---: | ---: | ---: | ---: | ---: |
| 9,000 | 13,913,184 B | 3,947,120 B | 252 / 9 ms | 521 / 74 ms | 11,889,002
B |
| 59,000 | 80,505,856 B | 4,188,128 B | 607 / 41 ms | 1,033 / 318 ms |
77,939,002 B |
At 59,000 rows, retained heap was **94.8% lower**, with slower loading and
emission than the in-memory baseline. A separate 500-row fast-path check
produced no spill files. An earlier disk-prototype comparison reduced logical
file bytes by 43.6%; that comparison is not against the original in-memory
implementation.
These are local GC-delta retained-heap and logical-file-length measurements
on a shared machine, not peak heap/RSS, filesystem block usage, production
throughput or an OOM guarantee. The controlled JDBC benchmark is separate from
live engine tests; its private measurement harness is not included in this
patch.
#### Operational limits
- Temporary storage must be writable and have sufficient space. Disk history
grows with updates until split cleanup; no compaction or disk quota is
introduced.
- The admission threshold is not an exact JVM heap cap. Individual records,
schemas, configured queues and other task state still consume heap.
- Disk I/O can increase snapshot duration and checkpoint latency. Atomic
emission still holds the checkpoint lock.
- Normal completion/failure/cancellation closes owned files. Abrupt process
termination may leave files for host/container lifecycle cleanup.
### How was this patch tested?
- CDC-base suite: **141 tests passed on Java 11 and Java 17**, with zero
failures/errors/skips. The Java 11 suite and scoped Spotless check were rerun
successfully on the final rebased commit. Regression coverage includes
reconciliation order and values, spill transitions, emission failure cleanup,
cancellation and corrupted record lengths. The allocation-guard regression
failed before its fix and passed afterward.
- Java 11 and Java 17 affected-reactor package builds passed; the Java 11
reactor included 76 modules and MySQL E2E test compilation. The Java 17 E2E
test-compile phase was up to date, not a fresh E2E compilation.
- **Five real Zeta/MySQL Java 11 E2E methods passed**: the new
snapshot-emission checkpoint/restore test, the existing exactly-once
multi-table/custom-key smoke test and three existing restore methods.
- The new `testMysqlCdcExactlyOnceSnapshotEmissionCheckpointRestore` uses
5,000 rows, positively observes spill storage and a checkpoint blocked on the
emitter-owned monitor, completes a savepoint, validates exact row multiplicity,
then restores and delivers row 5,001 without snapshot replay. Storage cleanup
is checked after completion and cancellation.
- Real-MySQL adapter checks covered terminal-range growth, checkpoint during
reconciliation and injected emission failure/restore with modeled sink
rollback. These are separate from the engine E2E tests.
- **24/24 defensive storage checks passed across Java 11/17**, including
corrupt lengths/offsets, partial writes, cleanup failures and POSIX
permissions. Independent correctness/security reviews and scoped
formatting/diff checks completed.
Focused unit suite:
```sh
mvn -pl seatunnel-connectors-v2/connector-cdc/connector-cdc-base test
```
Engine validation demonstrates completed-savepoint restore after a
checkpoint request during emission. It does **not** establish
crash-mid-emission or reconciliation-phase engine fault recovery. Runtime
validation covered MySQL/Zeta; the original Flink 1.18/Java 8 incident, all six
database runtimes, native Windows permissions and hosted CI have not been
validated by these local runs. Current dev targets Java 11. The earlier
engine/build measurements were made before the non-overlapping API-only #12807
upstream commit; the final branch is rebased onto it with identical CDC source
hashes.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [x] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
Checklist notes: no new connector, dependency, binary, option or module;
plugin mappings, distribution packaging, CI scopes and plugin configuration
need no changes. EN/ZH documentation and MySQL E2E coverage are included. Disk
requirements and latency tradeoffs are documented; no incompatible
API/configuration/checkpoint-format change is introduced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
