[ 
https://issues.apache.org/jira/browse/CAMEL-27295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Claus Ibsen updated CAMEL-27295:
--------------------------------
    Fix Version/s: 4.25.0
                       (was: 4.24.0)

> camel-azure-storage-blob - get each blob just before routing it
> ---------------------------------------------------------------
>
>                 Key: CAMEL-27295
>                 URL: https://issues.apache.org/jira/browse/CAMEL-27295
>             Project: Camel
>          Issue Type: Improvement
>          Components: camel-azure
>            Reporter: Claus Ibsen
>            Priority: Major
>             Fix For: 4.25.0
>
>
> Follow-up of CAMEL-26655 (PR https://github.com/apache/camel/pull/30481).
> h3. Where CAMEL-26655 leaves it
> PR #30481 adds {{maxMessagesPerPoll}} and lists the container page by page, 
> but it still gets every blob of the poll (opens the stream, or downloads to 
> {{fileDir}}) before the first exchange is routed. That keeps 
> {{CamelBatchSize}}, {{CamelBatchIndex}} and {{CamelBatchComplete}} exact. 
> With {{maxMessagesPerPoll}} set, this is bounded by the limit. With the 
> default {{maxMessagesPerPoll=0}}, every listed blob is still opened up front: 
> the cost measured in CAMEL-26655 (48 MiB buffered before the first exchange 
> for 12 blobs of 5 MiB).
> h3. Proposed change
> * Get each blob just before its exchange is routed, fetching one blob ahead, 
> as the S3 consumer does since CAMEL-26662 and minio since CAMEL-17100.
> * Keep {{maxMessagesPerPoll}} counting blobs that were got, not listed ones 
> (the CAMEL-25770 rule: blobs that cannot be got, are in progress or are gone 
> do not count).
> * Keep the CAMEL-25770 rule that a poll fails only when no listed blob could 
> be got.
> * Batch properties: with one blob fetched ahead, {{CamelBatchIndex}} and 
> {{CamelBatchComplete}} stay exact (an exchange is the last one when there is 
> no next successful get). Only {{CamelBatchSize}} is no longer exact, since 
> the number of successful gets is not known when the first exchange is routed.
> h3. Docs and upgrade guide
> * Document the batch properties, and that the {{completionFromBatchConsumer}} 
> option of the Aggregate EIP does not fit this consumer 
> ({{AggregateProcessor}} counts exchanges against {{CamelBatchSize}}).
> * Add an entry to the 4.24 upgrade guide: blobs are read at routing time 
> instead of at poll time, and {{CamelBatchSize}} changes meaning.
> _Claude Code on behalf of Claus Ibsen_



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to