Mani created CAMEL-24575:
----------------------------
Summary: SFTP Consumer: Request for lock-strategy-agnostic
minimum-age file filtering
Key: CAMEL-24575
URL: https://issues.apache.org/jira/browse/CAMEL-24575
Project: Camel
Issue Type: Improvement
Components: camel-ftp, camel-sftp
Affects Versions: 4.14.5
Reporter: Mani
*Background:*
CAMEL-23931 introduced preSort=modified which resolved our oldest-first
ordering requirement beautifully. Thank you for the quick turnaround on that.
This ticket is a follow-up for a related gap we are still working around.
*Our Use Case:*
We need the SFTP consumer to handle two guarantees when picking up files:
* Process files in oldest-modified-first order — resolved by preSort=modified
* Skip files that are still actively being written to the server — only pick
up files that have been stable for a minimum duration
*The Gap:*
Camel already has readLockMinAge which is exactly the semantic we need for
requirement 2. However, in our setup we use a custom lock strategy with
readLock=none and readLockMinAge appears to be unavailable unless
readLock=changed is configured. This means we cannot benefit from it regardless
of what we set.
*Current Workaround:*
We maintain a size-snapshot tracker inside our custom lock strategy
implementation that records file sizes across consecutive poll cycles and only
allows processing once the size has been unchanged for the configured duration.
*The Ask:*
Would it be feasible to expose a min-age filter at the same level as preSort —
operating on the SftpRemoteFile[] array before any lock strategy is involved?
Since this sits upstream of lock acquisition it would be lock-strategy-agnostic
and works alongside any custom locking. Combined with preSort=modified this
would give us oldest-stable-file-first natively and let us remove all custom
stability-tracking.
*References:*
CAMEL-23931 — preSort feature (related prior request)
Mailing list thread ([email protected], August 12):
[link|https://lists.apache.org/thread/3nwv3rh5ffjl5v5x08hprq3w0qckn28o]
Tagging [~davsclaus] for visibility
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
