0AyanamiRei opened a new pull request, #67874:
URL: https://github.com/apache/doris/pull/67874
### What problem does this PR solve?
Issue Number: N/A
Related PR: #67869
Problem Summary:
Amazon Kinesis GetRecords is a non-blocking pull API. An empty Records array
does not mean that the shard has reached the stream tip: the response can
still contain a non-empty NextShardIterator, and a later call using that
iterator can return records. The only terminal signal for a closed shard is
an empty NextShardIterator.
Routine Load Kinesis handled the response as follows:
1. KinesisDataConsumer::group_consume() called GetRecords.
2. It saved the returned NextShardIterator.
3. When Records was empty, it erased the shard from _consuming_shard_ids.
4. If this was the only active shard, the loop marked the consumer done,
shut down the queue, and returned successfully.
5. The next task recreated an iterator from the old committed sequence, so it
could observe the same empty page again instead of following the iterator
returned by Kinesis.
This can leave a shard stuck without consuming records that arrive after the
empty response. It also makes the result depend on whether the task happens
to have another active shard.
This change keeps the shard in the active set when Records is empty and
NextShardIterator is non-empty. The next polling round therefore follows the
returned iterator. The existing empty-iterator branch is unchanged and
continues to remove the closed shard only after the existing queue ordering
and end-of-shard handling. Queue shutdown, retry handling, transaction
attachments, and FE progress formats are unchanged.
### Reproduction
The new BE unit test invokes the production
KinesisDataConsumer::group_consume() method with an AWS SDK KinesisClient
fake:
- response 1: no records, MillisBehindLatest = 100, iterator
iterator-after-empty-page;
- response 2: one record, MillisBehindLatest = 0, empty iterator.
Before this fix, the test failed because the consumer made only one
GetRecords call:
Expected: 2
Actual: 1
After this fix, it verifies both that the second call is made and that it
uses iterator-after-empty-page. A second test covers two consecutive empty
responses before the record.
### Release note
Fix Kinesis Routine Load polling after empty GetRecords responses so later
records in the same shard are not skipped or left permanently unconsumed.
### Check List (For Author)
- Test: Unit Test
- ./run-be-ut.sh --run --filter='KinesisEmptyPageReproduction.*' -j48
failed before the fix and passed 2/2 after the fix.
- ./run-be-ut.sh --run
--filter='KinesisBatchProgressReproduction.*:KinesisEmptyPageReproduction.*'
-j48
passed 7/7 after the fix, including the existing Kinesis progress
boundary tests from the base branch.
- build-support/check-build-hygiene.sh passed.
- build-support/check-format.sh passed.
- git diff --check passed.
- The new test's clang-tidy check passed. The production file's full
clang-tidy check still reports pre-existing diagnostics outside this
change, including existing complexity and const-reference warnings.
- Behavior changed: Yes. Empty non-terminal Kinesis responses are polled
again using the returned iterator.
- Does this need documentation: No.
### Scope and limitations
This PR addresses only empty non-terminal GetRecords responses. It does not
change shard discovery, reshard parent/child ordering, LATEST initialization,
or FE shard-topology persistence. Live AWS ingestion was not used because the
machine has multiple shared Doris processes and no verifiable cluster
identity
for this workspace; the deterministic unit test exercises the production BE
consumer and AWS SDK response path without creating external resources.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
