Mani created CAMEL-23931:
----------------------------
Summary: Apache Camel SFTP Consumer: pre-sort by last-modified
time without creating Exchange objects for all files on server
Key: CAMEL-23931
URL: https://issues.apache.org/jira/browse/CAMEL-23931
Project: Camel
Issue Type: Improvement
Components: camel-ftp, camel-sftp
Affects Versions: 4.14.5
Reporter: Mani
Fix For: Future
We have a requirement to consume the oldest stable files first from an SFTP
server using maxMessagesPerPoll=10. We found that the two available
configurations both have trade-offs:
* eagerMaxMessagesPerPoll=false + sortBy=file:modified — achieves oldest-first
correctly, but creates Exchange/GenericFileMessage objects for every file on
the server before sorting, which doesn't scale with large directories.
* eagerMaxMessagesPerPoll=true (default) + sortBy=file:modified — limits to 10
files before sorting, breaking the oldest-first guarantee.
To achieve sort-before-limit without the memory overhead, we extended
SftpOperations.listFiles() to sort the raw SftpRemoteFile[] array by mtime
ascending immediately after channel.ls() — before any RemoteFile,
DefaultExchange, or GenericFileMessage objects are created. The full approach
with CustomSortedSftpComponent, CustomSortedSftpEndpoint, and
CustomSortedSftpOperations is documented in the Stack Overflow question below,
along with the two questions we're looking to get community input on.
Stack Overflow:
[(link)|https://stackoverflow.com/questions/79962276/apache-camel-sftp-consumer-pre-sort-by-last-modified-time-without-creating-exch]
Mailing list thread ([email protected], June 29):
[(link)|https://lists.apache.org/thread/czjjlxk6b948mzqwp2oqwvhls83jykbd]
Tagging [~davsclaus] for visibility
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
