[
https://issues.apache.org/jira/browse/CAMEL-25558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-25558:
--------------------------------
Fix Version/s: 4.24.0
> camel-aws2-s3 - a listed object that cannot be got (deleted meanwhile, or a
> failed get) fails the poll and stays in the in-progress repository, so it is
> never consumed until a restart
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25558
> URL: https://issues.apache.org/jira/browse/CAMEL-25558
> Project: Camel
> Issue Type: Bug
> Components: camel-aws2-s3
> Reporter: shashank
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
>
> When the consumer lists the bucket, {{AWS2S3Consumer.createExchanges(List)}}
> adds each listed key to the endpoint's {{inProgressRepository}} (line 264 at
> main 3a6cdba883a5) and then gets the object (line 287). When that get throws,
> the catch block (lines 301-315) closes the streams and removes the keys of
> the exchanges that were already created, and rethrows, so the poll fails. The
> key of the object that failed has no exchange yet, so it is not removed. The
> default repository is a memory repository that is only cleared when the
> endpoint stops, so every later poll skips that key as "already in progress"
> (lines 264-268).
> Effects:
> * A listed object that is deleted before the get (another consumer on the
> same bucket, a lifecycle rule, or the exchange of an earlier poll that
> completes after the listing, with an asynchronous route) makes the get fail
> with {{NoSuchKeyException}}: that poll fails. The other listed objects are
> consumed by a following poll (with a truncated listing the marker has already
> moved on, lines 173-188, so they wait until the listing starts over). A new
> object stored later under the same key is never consumed.
> * An object whose get fails for another reason (403, throttling, a network
> error), or whose download fails while it is read into the body
> ({{includeBody=true}}, line 500), fails that poll in the same way, stays in
> the bucket, and is never consumed again, until the endpoint is restarted. In
> the second case the exchange that was created for it (line 493) is also never
> released to the exchange factory.
> This is not new and not caused by CAMEL-25521: camel-4.22.x has the same loop
> and catch block.
> For comparison: camel-minio (CAMEL-25512) and camel-google-storage
> (CAMEL-25519) skip a listed object that no longer exists (DEBUG) and consume
> the rest; camel-ibm-cos skips a listed object whose get fails for any reason
> (WARN with the key and the message, no stack trace), removes its key and
> continues with the rest.
> h3. Reproduction
> New {{AWS2S3ConsumerDeletedObjectTest}} (6 tests) with the in-process
> {{FakeS3Server}} of the module (it can now remove an object right after a
> listing returned it, refuse the gets of an object with 403, and cut the body
> of a get short), and {{AWS2S3ConsumerDeletedObjectPooledExchangeTest}}, which
> runs the same 6 with {{PooledExchangeFactory}} and adds one. The scheduler is
> not started ({{startScheduler=false}}); the tests call {{poll()}}. On main
> all 13 fail:
> * {{listedObjectThatNoLongerExistsIsSkipped}}: {{a.txt}}, {{b.txt}},
> {{c.txt}}; {{b.txt}} is removed after the listing. Expected: the poll returns
> 2 and routes a and c with {{CamelBatchSize}} 2. On main:
> {{NoSuchKeyException: The specified key does not exist}} from {{poll()}}.
> * {{objectCreatedAgainAfterItWasSkippedIsConsumed}}: after that poll a new
> {{b.txt}} is stored and must be consumed by the next poll. On main:
> {{NoSuchKeyException}} from the first poll.
> * {{objectWhoseGetIsRefusedDoesNotStopTheOtherObjects}}: every get of
> {{b.txt}} is refused. Expected: the poll returns 2 (a and c), the next poll
> consumes a new {{d.txt}} and tries {{b.txt}} again, and {{b.txt}} is consumed
> once its get is allowed. On main: {{S3Exception: Access Denied}} from the
> first {{poll()}}.
> * {{skippedObjectOfATruncatedListingIsTriedAgainWhenTheListingStartsOver}}
> ({{maxMessagesPerPoll=2}}, four objects, {{b.txt}} refused): the polls return
> 1, 2 (second page, {{b.txt}} not tried), 0 (listing starts over, {{b.txt}}
> tried again), then 1 once the get is allowed. On main: {{S3Exception}} from
> the first {{poll()}}.
> * {{objectWhoseDownloadFailsIsSkippedAndNotLeftInProgress}}: the content
> stream throws an {{IOException}} while it is read into the body. Expected:
> the poll returns 0, the stream is closed, the key is not in progress. On
> main: {{RuntimeCamelException}} ({{IOException: connection reset}}) from
> {{poll()}}.
> * {{objectStreamIsClosedWhenIncludeS3ObjectFails}}: an overridden
> {{includeS3Object}} throws. Expected as above. On main:
> {{IllegalStateException}} from {{poll()}}.
> * {{exchangeOfAnObjectWhoseDownloadFailsIsGivenBackToThePool}} (pooled): the
> body of {{b.txt}} is cut short. Expected: a, c and d are routed with their
> own bodies, 4 exchanges taken from the pool and 4 released. On main:
> {{RuntimeCamelException}} ({{ConnectionClosedException: Premature end of
> Content-Length delimited message body}}) from {{poll()}}.
> h3. Proposed fix
> In {{createExchanges(List)}}: a failure to get one listed object, or to
> create its exchange, no longer fails the poll. The object is skipped (stream
> closed, exchange given back to the exchange factory, key removed from the
> in-progress repository), the other listed objects are consumed in the same
> poll, the skipped object is not counted, and a later poll tries it again.
> {{NoSuchKeyException}} is logged at DEBUG; any other failure at WARN with the
> key and the message, without a stack trace, because an object that can never
> be read (archived, access denied) is logged at every poll that lists it. This
> is what camel-ibm-cos does; the alternative (keep failing the poll and only
> remove the key) would make such an object fail every poll and block the
> objects listed after it.
> Consequence: such a failure is no longer a failed poll, so it does not reach
> the exception handler ({{bridgeErrorHandler}}) and does not count for
> {{backoffErrorThreshold}}. The listing marker is set from the listing
> response before the gets, as before: with a truncated listing a skipped
> object is tried again when the listing starts over, otherwise at the next
> poll.
> Not changed, known limits: with {{fileName}}, a failed get still fails the
> poll. An {{Error}} (for example {{OutOfMemoryError}} while a large object is
> read into the body) is not caught, so its key still stays in progress.
> With the pooled exchange factory the exchange of a skipped object is given
> back with {{done()}}: {{releaseExchange(exchange, true)}} on an exchange that
> is not done adds it to the pool twice, and two later objects of the poll then
> share it (seen with the pooled test). The same call for never-routed
> exchanges exists on main in this consumer ({{processBatch}} when stopping,
> the outer catch) and in other consumers; not changed here.
> No new option. 4.23 upgrade-guide note under the existing camel-aws2-s3
> in-progress entry. camel-aws2-s3 unit tests with the fix: 98, 0 failures (the
> 68 integration tests need LocalStack and were not run).
> Found by comparing the consumer with the other object storage consumers after
> CAMEL-25521, then confirmed with the real consumer as above.
> Affected: main; camel-4.22.x, camel-4.18.x and camel-4.14.x have the same
> loop and catch block (read, not run).
> Duplicate check (2026-10-11): JIRA text "aws2-s3 NoSuchKey" (1: CAMEL-25521,
> the related fix, which does not change this path), "AWS2S3Consumer
> NoSuchKeyException" (1: CAMEL-17110, which added the in-progress repository),
> component camel-aws2-s3 with text "inProgressRepository" (1: CAMEL-25521),
> component camel-aws2-s3 created since 2026-01-01 (10 issues, none about a
> deleted object or a failed get in the consumer); GitHub pull requests
> "AWS2S3Consumer NoSuchKey" and "aws2-s3 NoSuchKey": none; no open pull
> request touches {{AWS2S3Consumer}}.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
