spoorthibasu commented on code in PR #4550:
URL: https://github.com/apache/flink-cdc/pull/4550#discussion_r4056979303
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-cdc-base/src/main/java/org/apache/flink/cdc/connectors/base/source/reader/IncrementalSourceReader.java:
##########
@@ -481,6 +501,14 @@ private void requestStreamSplitMetaIfNeeded(StreamSplit
streamSplit) {
} else {
LOG.info("The meta of stream split {} has been collected success",
splitId);
this.addSplits(Collections.singletonList(streamSplit));
+ // Report the stream-split metadata assembled so the coordinator
can release what it no
+ // longer needs to serve. Carries the assignment generation so a
stale report from a
+ // failed attempt is ignored.
+ context.sendSourceEventToCoordinator(
Review Comment:
Removed the duplicate send, and the comment in the completed-split branch
now mentions the group-fetch case along with the others that reach it.
##########
docs/content/docs/connectors/flink-sources/db2-cdc.md:
##########
@@ -263,6 +263,13 @@ Db2 server.
If the flink version is greater than or equal to 1.15, the default
value of 'execution.checkpointing.checkpoints-after-tasks-finish.enabled' has
been changed to true,
so it does not need to be explicitly configured
'execution.checkpointing.checkpoints-after-tasks-finish.enabled' = 'true'
</td>
+ </tr>
+ <tr>
+ <td>scan.incremental.snapshot.metadata.release.enabled</td>
+ <td>optional</td>
+ <td style="word-wrap: break-word;">false</td>
+ <td>Boolean</td>
+ <td>Whether to release the snapshot split metadata (assigned splits,
finished offsets and table schemas) held by the source coordinator once the
source has entered the stream phase, to reduce JobManager memory on jobs with a
very large number of snapshot splits. Disabled by default. Incompatible with
scan.newly-added-table.enabled: enabling both fails at startup, and a job that
has released the metadata cannot later enable newly-added-table scanning.
Release happens only after a successful checkpoint; if checkpointing is
disabled or no checkpoint completes, the metadata is retained, so this option
has no effect without checkpointing.</td>
Review Comment:
Documented for the four dialects and for MySQL, in both the Flink Source and
Pipeline Connector docs, EN and ZH.
The Db2 docs also drop the `scan.newly-added-table.enabled` line, since Db2
doesn't expose that option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
