[
https://issues.apache.org/jira/browse/CAMEL-23931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen reassigned CAMEL-23931:
-----------------------------------
Assignee: Claus Ibsen
> Apache Camel SFTP Consumer: pre-sort by last-modified time without creating
> Exchange objects for all files on server
> --------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-23931
> URL: https://issues.apache.org/jira/browse/CAMEL-23931
> Project: Camel
> Issue Type: Improvement
> Components: camel-ftp, camel-sftp
> Affects Versions: 4.14.5
> Reporter: Mani
> Assignee: Claus Ibsen
> Priority: Major
>
> We have a requirement to consume the oldest stable files first from an SFTP
> server using maxMessagesPerPoll=10. We found that the two available
> configurations both have trade-offs:
> * eagerMaxMessagesPerPoll=false + sortBy=file:modified — achieves
> oldest-first correctly, but creates Exchange/GenericFileMessage objects for
> every file on the server before sorting, which doesn't scale with large
> directories.
> * eagerMaxMessagesPerPoll=true (default) + sortBy=file:modified — limits to
> 10 files before sorting, breaking the oldest-first guarantee.
> To achieve sort-before-limit without the memory overhead, we extended
> SftpOperations.listFiles() to sort the raw SftpRemoteFile[] array by mtime
> ascending immediately after channel.ls() — before any RemoteFile,
> DefaultExchange, or GenericFileMessage objects are created. The full approach
> with CustomSortedSftpComponent, CustomSortedSftpEndpoint, and
> CustomSortedSftpOperations is documented in the Stack Overflow question
> below, along with the two questions we're looking to get community input on.
> Stack Overflow:
> [(link)|https://stackoverflow.com/questions/79962276/apache-camel-sftp-consumer-pre-sort-by-last-modified-time-without-creating-exch]
>
> Mailing list thread ([email protected], June 29):
> [(link)|https://lists.apache.org/thread/czjjlxk6b948mzqwp2oqwvhls83jykbd]
>
> Tagging [~davsclaus] for visibility
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
