sun-9545sunoj opened a new pull request, #12463:
URL: https://github.com/apache/seatunnel/pull/12463
### Purpose of this pull request
Fixes #10270
During the CDC full snapshot stage (startup.mode = "initial"), if the
table's primary key IDs are highly non-consecutive (e.g., large ranges of
deleted rows resulting in sparse clusters of data), the job can fail with
CheckpointException: Checkpoint expired before completing.
Root Cause: In AbstractJdbcSourceChunkSplitter.splitTableIntoChunks(), the
distributionFactor is calculated as (MAX - MIN + 1) / rowCount. If this factor
falls within [0.05, 100.0], the data is deemed "evenly distributed" and split
EvenlySizedChunks() is invoked. However, this method uses pure arithmetic
stepping (min + step, min + 2*step). When IDs are clustered with large gaps,
some arithmetic chunks end up empty while others hold a disproportionately
large number of rows, causing the reader task for those overloaded chunks to
exceed the checkpoint timeout.
Fix: When distributionFactor > 1.0 (which mathematically guarantees gaps
exist in the ID space), the chunk splitter now prefers
efficientShardingThroughSampling(). Sampling uses actual data points to
determine chunk boundaries, ensuring splits remain balanced regardless of gap
distribution. It gracefully falls back to arithmetic splitting if sampling is
disabled (sampleShardingAllow = false)or returns no data.
### Does this PR introduce any user-facing change?
No. Configuration options remain unchanged. This strictly improves the
internal chunk-splitting robustness for sparse tables.
### How was this patch tested?
• Added testSplitEvenlySizedChunksWithNonConsecutiveIds in
AbstractJdbcSourceChunkSplitterTest to verify that:
• Sampling-based splitting is selected when IDs are non-consecutive
(distributionFactor > 1.0).
• Arithmetic splitting is retained when IDs are consecutive
(distributionFactor <= 1.0) or when sampling is explicitly disabled.
• Ran mvn spotless:apply and mvn -q -DskipTests verify.
• All tests in AbstractJdbcSourceChunkSplitterTest passed locally.
### Check list
[ +] If any new Jar binary package adding in your PR, please add License
Notice according New License Guide
https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md
[ +] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
[+ ] If necessary, please update incompatible-changes.md to describe the
incompatibility caused by this PR.
[ +] If you are contributing the connector code, please check that the
following files are updated:
1. Update plugin-mapping.properties
https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties and add
new connector information in it
2. Update the pom file of seatunnel-dist
https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml
3. Add ci label in label-scope-conf
https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml
5. Add e2e testcase in seatunnel-e2e
https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/
6. Update connector plugin_config
https://github.com/apache/seatunnel/blob/dev/config/plugin_config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
