[
https://issues.apache.org/jira/browse/CAMEL-26653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen resolved CAMEL-26653.
---------------------------------
Resolution: Fixed
Fixed by https://github.com/apache/camel/pull/29656 (merged to main for
4.24.0). _Claude Code on behalf of davsclaus_
> camel-minio - maxMessagesPerPoll only sets the listing page size: one poll
> lists, stats and routes the whole bucket
> -------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26653
> URL: https://issues.apache.org/jira/browse/CAMEL-26653
> Project: Camel
> Issue Type: Bug
> Components: camel-minio
> Affects Versions: 4.14.9, 4.18.5, 4.22.2, 4.23.0
> Reporter: Federico Mariani
> Priority: Minor
> Fix For: 4.24.0
>
>
> h3. Problem
> The {{maxMessagesPerPoll}} option (default 10) is documented as "the maximum
> number of messages as a limit to poll at each polling ... Use 0 or a negative
> number to set it as unlimited". The minio consumer uses it only as the page
> size of the listing, so one poll lists every page of the bucket, gets the
> metadata of every object, and routes all of them.
> Code on main (commit 7978db3d42dd), in
> {{components/camel-minio/src/main/java/org/apache/camel/component/minio/MinioConsumer.java}}:
> * Lines 156-158: {{listObjectRequest.maxKeys(maxMessagesPerPoll)}}.
> * Lines 226-242: {{createExchanges}} loops with {{do ... while
> (minioObjectSummaries.hasNext())}} and has no count check.
> * In minio-java 9.0.3, the iterator returned by {{listObjects}} is lazy:
> when a page is truncated, {{hasNext()}} and {{next()}} fetch the next one
> ({{MinioAsyncClient.ObjectIterator}} calls {{populate()}} again).
> * So the loop reads every page.
> * Line 567: each listed object gets a {{statObject}} before anything is
> routed ({{MinioEndpoint.getObjectStat}}, {{MinioEndpoint.java}} line 246).
> * Line 346: {{processBatch}} routes every exchange and has no limit either.
> With the defaults ({{includeBody=true}}, {{deleteAfterRead=true}}), each
> object is downloaded and then deleted.
> * Lines 191-195: the continuation check compares the number of objects listed
> by the whole poll to {{maxMessagesPerPoll}}. For any bucket with more objects
> than that, the next poll starts after the last object of the bucket and gets
> an empty listing.
> The same behaviour is in camel-4.14.9, 4.18.5, 4.22.2 and 4.23.0 (checked in
> the source, which is structured slightly differently there).
> From the code, for 10k objects with the default of 10: about 1000 list
> requests and 10k stat requests happen before the first exchange is routed,
> and that one poll then routes all 10k objects. This figure is arithmetic, not
> measured at that size.
> h3. How to reproduce
> Measured on main with a JUnit test:
> * A real {{MinioClient}} against an in-process fake S3 endpoint: the module's
> {{FakeS3Server}} test helper, extended with request counters. It pages
> ListObjectsV2 by max-keys and continuation-token.
> * 30 objects, {{maxMessagesPerPoll=10}}, {{startScheduler=false}}, polls
> called by the test.
> ||deleteAfterRead||poll||list requests||stat (HEAD)||get||routed||
> |true|1|3|30|30|30|
> |true|2|1, start-after=o030.txt, empty|0|0|0|
> |false|1|3|30|30|30|
> |false|2|1, empty|0|0|0|
> |false|3|3|30|30|30|
> With the documented limit, a poll would make 1 list request and route at most
> 10 objects.
> h3. Proposed change
> * When {{maxMessagesPerPoll}} is above 0, stop reading the listing once that
> many objects were listed. That means one page per poll, as camel-aws2-s3
> does: {{AWS2S3Consumer}} makes one ListObjectsV2 call with {{maxKeys}} and
> keeps the continuation token for the next poll.
> * Base the continuation on what this poll listed: continue after the last
> listed object when the page was full, and start over when the listing has
> ended. This also removes the empty listing every other poll.
> * Keep {{maxMessagesPerPoll}} 0 or below as today (all pages).
> * Risk: some listed objects are not routed. These are folders (when
> {{includeFolders=false}}), objects still in flight (CAMEL-25512) and objects
> that cannot be got (CAMEL-25770). So a poll can route fewer objects than the
> limit, and the next poll continues after them, as with aws2-s3.
> h3. Impact / upgrade
> * Filed as a Bug: the documented option is not applied to routing, which goes
> beyond a performance issue.
> * This is a behaviour change. With the default {{maxMessagesPerPoll=10}}, a
> poll routes at most 10 objects instead of the whole bucket, so draining a
> large bucket takes more polls. Setting {{maxMessagesPerPoll=0}} keeps the old
> behaviour.
> * It needs an entry in the 4.24 upgrade guide.
> * An endpoint with {{objectName}} set gets one object per poll and is not
> affected.
> * The component has been deprecated since 4.23 (CAMEL-24750). The change is
> small and the option is documented, so it still seems worth fixing.
> h3. Related
> * CAMEL-17100: "Minio Consumer is really slow on listing initially if you
> have a lot of files". Its fix (7de327d91eb6) moved the download of each
> object from the listing into {{processBatch}}. The listing over all pages and
> the stat per object stayed.
> * Commit 62fd4017653d introduced the continuation check that compares the
> count of the whole poll.
> * The loop without a count check has been there since the component was added
> (67bad5a5978a, CAMEL-13934).
> * CAMEL-25512 and CAMEL-25770: recent fixes in the same loop.
> _Claude Code on behalf of Federico Mariani_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
