[
https://issues.apache.org/jira/browse/CAMEL-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264336#comment-16264336
]
ASF GitHub Bot commented on CAMEL-11656:
----------------------------------------
onders86 closed pull request #2108: CAMEL-11656 - add preSort option
URL: https://github.com/apache/camel/pull/2108
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/camel-core/src/main/docs/file-component.adoc
b/camel-core/src/main/docs/file-component.adoc
index 8aa0247103e..294ced12221 100644
--- a/camel-core/src/main/docs/file-component.adoc
+++ b/camel-core/src/main/docs/file-component.adoc
@@ -69,7 +69,7 @@ with the following path and query parameters:
| *directoryName* | *Required* The starting directory | | File
|===
-==== Query Parameters (80 parameters):
+==== Query Parameters (81 parameters):
[width="100%",cols="2,5,^1,2",options="header"]
|===
@@ -82,6 +82,7 @@ with the following path and query parameters:
| *moveFailed* (consumer) | Sets the move failure expression based on Simple
language. For example to move files into a .error subdirectory use: .error.
Note: When moving the files to the fail location Camel will handle the error
and will not pick up the file again. | | String
| *noop* (consumer) | If true the file is not moved or deleted in any way.
This option is good for readonly data or for ETL type requirements. If
noop=true Camel will set idempotent=true as well to avoid consuming the same
files over and over again. | false | boolean
| *preMove* (consumer) | Expression (such as File Language) used to
dynamically set the filename when moving it before processing. For example to
move in-progress files into the order directory set this value to order. | |
String
+| *preSort* (consumer) | When pre-sort is enabled then the consumer will sort
the file and directory names during polling that was retrieved from the file
system. You may want to do this in case you need to operate on the files in a
sorted order. The pre-sort is executed before the consumer starts to filter and
accept files to process by Camel. This option is default=false meaning
disabled. | false | boolean
| *recursive* (consumer) | If a directory will look for files in all the
sub-directories as well. | false | boolean
| *sendEmptyMessageWhenIdle* (consumer) | If the polling consumer did not poll
any files you can enable this option to send an empty message (no body)
instead. | false | boolean
| *directoryMustExist* (consumer) | Similar to startingDirectoryMustExist but
this applies during polling recursive sub directories. | false | boolean
diff --git
a/camel-core/src/main/java/org/apache/camel/component/file/FileConsumer.java
b/camel-core/src/main/java/org/apache/camel/component/file/FileConsumer.java
index 5e3a180bc50..aa3dcfafc2e 100644
--- a/camel-core/src/main/java/org/apache/camel/component/file/FileConsumer.java
+++ b/camel-core/src/main/java/org/apache/camel/component/file/FileConsumer.java
@@ -21,11 +21,13 @@
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Arrays;
+import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.stream.Collectors;
import org.apache.camel.Exchange;
import org.apache.camel.Message;
@@ -84,6 +86,9 @@ protected boolean pollDirectory(String fileName,
List<GenericFile<File>> fileLis
}
}
List<File> files = Arrays.asList(dirFiles);
+ if (getEndpoint().isPreSort()) {
+ Collections.sort(files, (a, b) ->
a.getAbsoluteFile().compareTo(a.getAbsoluteFile()));
+ }
for (File file : dirFiles) {
// check if we can continue polling in files
diff --git
a/camel-core/src/main/java/org/apache/camel/component/file/GenericFileEndpoint.java
b/camel-core/src/main/java/org/apache/camel/component/file/GenericFileEndpoint.java
index ec4c0c5c6e6..cc7c7020689 100644
---
a/camel-core/src/main/java/org/apache/camel/component/file/GenericFileEndpoint.java
+++
b/camel-core/src/main/java/org/apache/camel/component/file/GenericFileEndpoint.java
@@ -111,6 +111,8 @@
protected boolean recursive;
@UriParam(label = "consumer")
protected boolean delete;
+ @UriParam(label = "consumer")
+ protected boolean preSort;
@UriParam(label = "consumer,filter")
protected int maxMessagesPerPoll;
@UriParam(label = "consumer,filter", defaultValue = "true")
@@ -414,6 +416,20 @@ public void setAntFilterCaseSensitive(boolean
antFilterCaseSensitive) {
public GenericFileFilter<T> getAntFilter() {
return antFilter;
}
+
+ public boolean isPreSort() {
+ return preSort;
+ }
+
+ /**
+ * When pre-sort is enabled then the consumer will sort the file and
directory names during polling,
+ * that was retrieved from the file system. You may want to do this in
case you need to operate on the files
+ * in a sorted order. The pre-sort is executed before the consumer starts
to filter, and accept files
+ * to process by Camel. This option is default=false meaning disabled.
+ */
+ public void setPreSort(boolean preSort) {
+ this.preSort = preSort;
+ }
public boolean isDelete() {
return delete;
diff --git a/components/camel-ftp/src/main/docs/ftp-component.adoc
b/components/camel-ftp/src/main/docs/ftp-component.adoc
index 3c9de248d92..97fa84b9d6d 100644
--- a/components/camel-ftp/src/main/docs/ftp-component.adoc
+++ b/components/camel-ftp/src/main/docs/ftp-component.adoc
@@ -97,7 +97,7 @@ with the following path and query parameters:
| *directoryName* | The starting directory | | String
|===
-==== Query Parameters (103 parameters):
+==== Query Parameters (104 parameters):
[width="100%",cols="2,5,^1,2",options="header"]
|===
@@ -115,6 +115,7 @@ with the following path and query parameters:
| *moveFailed* (consumer) | Sets the move failure expression based on Simple
language. For example to move files into a .error subdirectory use: .error.
Note: When moving the files to the fail location Camel will handle the error
and will not pick up the file again. | | String
| *noop* (consumer) | If true the file is not moved or deleted in any way.
This option is good for readonly data or for ETL type requirements. If
noop=true Camel will set idempotent=true as well to avoid consuming the same
files over and over again. | false | boolean
| *preMove* (consumer) | Expression (such as File Language) used to
dynamically set the filename when moving it before processing. For example to
move in-progress files into the order directory set this value to order. | |
String
+| *preSort* (consumer) | When pre-sort is enabled then the consumer will sort
the file and directory names during polling that was retrieved from the file
system. You may want to do this in case you need to operate on the files in a
sorted order. The pre-sort is executed before the consumer starts to filter and
accept files to process by Camel. This option is default=false meaning
disabled. | false | boolean
| *recursive* (consumer) | If a directory will look for files in all the
sub-directories as well. | false | boolean
| *sendEmptyMessageWhenIdle* (consumer) | If the polling consumer did not poll
any files you can enable this option to send an empty message (no body)
instead. | false | boolean
| *streamDownload* (consumer) | Sets the download method to use when not using
a local working directory. If set to true the remote files are streamed to the
route as they are read. When set to false the remote files are loaded into
memory before being sent into the route. | false | boolean
diff --git a/components/camel-ftp/src/main/docs/ftps-component.adoc
b/components/camel-ftp/src/main/docs/ftps-component.adoc
index 3c6260e3a2a..d01d6f96908 100644
--- a/components/camel-ftp/src/main/docs/ftps-component.adoc
+++ b/components/camel-ftp/src/main/docs/ftps-component.adoc
@@ -57,7 +57,7 @@ with the following path and query parameters:
| *directoryName* | The starting directory | | String
|===
-==== Query Parameters (111 parameters):
+==== Query Parameters (112 parameters):
[width="100%",cols="2,5,^1,2",options="header"]
|===
@@ -75,6 +75,7 @@ with the following path and query parameters:
| *moveFailed* (consumer) | Sets the move failure expression based on Simple
language. For example to move files into a .error subdirectory use: .error.
Note: When moving the files to the fail location Camel will handle the error
and will not pick up the file again. | | String
| *noop* (consumer) | If true the file is not moved or deleted in any way.
This option is good for readonly data or for ETL type requirements. If
noop=true Camel will set idempotent=true as well to avoid consuming the same
files over and over again. | false | boolean
| *preMove* (consumer) | Expression (such as File Language) used to
dynamically set the filename when moving it before processing. For example to
move in-progress files into the order directory set this value to order. | |
String
+| *preSort* (consumer) | When pre-sort is enabled then the consumer will sort
the file and directory names during polling that was retrieved from the file
system. You may want to do this in case you need to operate on the files in a
sorted order. The pre-sort is executed before the consumer starts to filter and
accept files to process by Camel. This option is default=false meaning
disabled. | false | boolean
| *recursive* (consumer) | If a directory will look for files in all the
sub-directories as well. | false | boolean
| *sendEmptyMessageWhenIdle* (consumer) | If the polling consumer did not poll
any files you can enable this option to send an empty message (no body)
instead. | false | boolean
| *streamDownload* (consumer) | Sets the download method to use when not using
a local working directory. If set to true the remote files are streamed to the
route as they are read. When set to false the remote files are loaded into
memory before being sent into the route. | false | boolean
diff --git a/components/camel-ftp/src/main/docs/sftp-component.adoc
b/components/camel-ftp/src/main/docs/sftp-component.adoc
index d00e65a276c..85122b108f0 100644
--- a/components/camel-ftp/src/main/docs/sftp-component.adoc
+++ b/components/camel-ftp/src/main/docs/sftp-component.adoc
@@ -48,7 +48,7 @@ with the following path and query parameters:
| *directoryName* | The starting directory | | String
|===
-==== Query Parameters (110 parameters):
+==== Query Parameters (111 parameters):
[width="100%",cols="2,5,^1,2",options="header"]
|===
@@ -65,6 +65,7 @@ with the following path and query parameters:
| *moveFailed* (consumer) | Sets the move failure expression based on Simple
language. For example to move files into a .error subdirectory use: .error.
Note: When moving the files to the fail location Camel will handle the error
and will not pick up the file again. | | String
| *noop* (consumer) | If true the file is not moved or deleted in any way.
This option is good for readonly data or for ETL type requirements. If
noop=true Camel will set idempotent=true as well to avoid consuming the same
files over and over again. | false | boolean
| *preMove* (consumer) | Expression (such as File Language) used to
dynamically set the filename when moving it before processing. For example to
move in-progress files into the order directory set this value to order. | |
String
+| *preSort* (consumer) | When pre-sort is enabled then the consumer will sort
the file and directory names during polling that was retrieved from the file
system. You may want to do this in case you need to operate on the files in a
sorted order. The pre-sort is executed before the consumer starts to filter and
accept files to process by Camel. This option is default=false meaning
disabled. | false | boolean
| *recursive* (consumer) | If a directory will look for files in all the
sub-directories as well. | false | boolean
| *sendEmptyMessageWhenIdle* (consumer) | If the polling consumer did not poll
any files you can enable this option to send an empty message (no body)
instead. | false | boolean
| *streamDownload* (consumer) | Sets the download method to use when not using
a local working directory. If set to true the remote files are streamed to the
route as they are read. When set to false the remote files are loaded into
memory before being sent into the route. | false | boolean
diff --git
a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpConsumer.java
b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpConsumer.java
index 1d72438e671..33f153da83d 100644
---
a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpConsumer.java
+++
b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpConsumer.java
@@ -17,7 +17,9 @@
package org.apache.camel.component.file.remote;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.List;
+import java.util.stream.Collectors;
import org.apache.camel.Exchange;
import org.apache.camel.Message;
@@ -143,6 +145,11 @@ protected boolean doPollDirectory(String absolutePath,
String dirName, List<Gene
// we found some files
log.trace("Found {} in directory: {}", files.size(), dir);
}
+
+
+ if (getEndpoint().isPreSort()) {
+ Collections.sort(files, (a, b) ->
a.getName().compareTo(b.getName()));
+ }
for (FTPFile file : files) {
diff --git
a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/SftpConsumer.java
b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/SftpConsumer.java
index 8eb3e23fc98..c7a4a4420bb 100644
---
a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/SftpConsumer.java
+++
b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/SftpConsumer.java
@@ -17,7 +17,9 @@
package org.apache.camel.component.file.remote;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.List;
+import java.util.stream.Collectors;
import com.jcraft.jsch.ChannelSftp;
import com.jcraft.jsch.SftpException;
@@ -142,6 +144,10 @@ protected boolean doPollDirectory(String absolutePath,
String dirName, List<Gene
// we found some files
log.trace("Found {} in directory: {}", files.size(), dir);
}
+
+ if (getEndpoint().isPreSort()) {
+ Collections.sort(files, (a, b) ->
a.getFilename().compareTo(b.getFilename()));
+ }
for (SftpRemoteFile file : files) {
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Support default directory sorter for FileConsumer
> -------------------------------------------------
>
> Key: CAMEL-11656
> URL: https://issues.apache.org/jira/browse/CAMEL-11656
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.19.0
> Reporter: Viktor Khoroshko
> Priority: Minor
> Fix For: 2.21.0
>
>
> Hello,
> The current way of sorting consumed files isn't very flexible due to how it's
> implemented:
> Files are sorted AFTER they're added to in progress repository what in
> combination with *maxMessagesPerPoll* set to be > 0 and
> *eagerLimitMaxMessagesPerPoll* set to false forces the
> *removeExcessiveInProgressFiles* call - which in case of persistent
> idempotent repository used causes redundant calls to a database.
> This is not only the issue.
> In my case I have a logic in a custom file filter that a file shouldn't be
> processed if a file with a same prefix is already in progress while still
> those files should be processed in a specified order.
> The issue is that since sorting is performed after each file is added to in
> progress repository and before added there a file should be accepted by a
> filter but it will be not - as a previous file with a same prefix has
> already been added to in progress repo.
> *Example*: files - test.001, test.002, test.003
> If test.003 is first in a returned file list then it will be added to in
> progress repository and next files will not be accepted due to the same
> prefix.
> Default sorting would solve this issue.
> It would be great if default sorter can be specified for *FileConsumer* in
> the pollDirectory method:
> {code:java}
> log.trace("Polling directory: {}", directory.getPath());
> File[] dirFiles = directory.listFiles();
> if (dirFiles == null || dirFiles.length == 0) {
> // no files in this directory to poll
> if (log.isTraceEnabled()) {
> log.trace("No files found in directory: {}",
> directory.getPath());
> }
> return true;
> } else {
> // we found some files
> if (log.isTraceEnabled()) {
> log.trace("Found {} in directory: {}", dirFiles.length,
> directory.getPath());
> }
> }
> List<File> files = Arrays.asList(dirFiles);
> // sort there?
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
