github-actions[bot] commented on code in PR #69363:
URL: https://github.com/apache/doris/pull/69363#discussion_r4343002299
##########
fe/fe-core/src/main/java/org/apache/doris/job/offset/s3/S3EventSourceOffsetProvider.java:
##########
@@ -573,10 +573,10 @@ private ParsedMessage parseMessage(Message message) {
if (isCommitted(key, message.messageId(),
message.receiptHandle())) {
continue;
}
+ String path = "s3://" + bucket + "/" + key;
files.add(key);
- // The object must remain unchanged until ingestion completes;
notifications identify keys, not versions.
- fileStatuses.add(new TBrokerFileStatus("s3://" + bucket + "/" +
key, false,
- record.s3.object.size, true));
+ fileStatuses.add(new TBrokerFileStatus(path, false,
record.s3.object.size, true)
+ .setModificationTime(System.currentTimeMillis()));
Review Comment:
[P1] Refresh the byte bound when invalidating an EVENT object's cache
With file caching and an explicit CSV schema, let a 4-byte invalid-UTF-8
object fail ingestion, overwrite it with `1,a\n2,b\n` (8 bytes), then RESUME.
If a receive contains the old notification before the replacement notification
([event ordering is not
guaranteed](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html)),
`receiveBatch` keeps the old size 4 for that key but records both receipts.
This new mtime bypasses the cached failed bytes; `S3FileReader` trusts the
nonnegative `file_size` and clips reads at byte 4, so the task successfully
inserts only `1,a\n` and acknowledges both events, losing the second row.
Before this change the same warm-cache retry kept failing on the old cached
bytes. Refresh the current size together with the cache identity (or reject a
changed object), and cover redelivery of the unchanged old notification; the
updated resume test instead supplies size 170 in a new notification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
