HuangZhenQiu commented on code in PR #18468:
URL: https://github.com/apache/hudi/pull/18468#discussion_r3036162561
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/reader/BatchRecords.java:
##########
@@ -80,6 +80,8 @@ public String nextSplit() {
public HoodieRecordWithPosition<T> nextRecordFromSplit() {
if (recordIterator.hasNext()) {
recordAndPosition.record(recordIterator.next());
+ // Mark the last record in the split so the emitter can emit a split-end
watermark.
+ recordAndPosition.setLastInSplit(!recordIterator.hasNext());
Review Comment:
In the current split handling logic, all of the data in one split will be
returned with a batch. After that an empty batch will be returned with a
completed split id to notify source framework the completeness of split. In the
empty batch, no record inside. We don't need to emit watermark for it. In the
handle with the idleness of some source reader, users can set watermark
strategy to periodical send watermark for source reader without data to process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
