Github user kl0u commented on the issue:
https://github.com/apache/flink/pull/2109
Hello @mushketyk and sorry for the late response.
Great that you are working on that also for the Batch API!
Recently we introduced in the Streaming API (not batch) the notion of
continuous file monitoring. In this context we also added the
```FileParhFilter``` class. As an example you can see the ```
readFile(FileInputFormat<OUT> inputFormat, String filePath, FileProcessingMode
watchType, long interval, FilePathFilter filter) ``` in the
```StreamExecutionEnvironment```.
What I would suggest in order to have this functionality for both batch and
streaming, is to remove it from a parameter in the configuration file, and pass
the ```FilePathFilter``` as an argument to the constructor of the
```FileInputFormat``` and then do the filtering the same way you do it. The
reason is:
1) Cleaner code, as we will not have 2 different ways to do the same thing
2) Better usability, as you can imagine a scenario where an administrator
sets a global path filter and then the user another one. In this case, which
should be respected?
3) Overloading the configuration file with job specific stuff is probably
not the best way to go.
This may also require some changes in the internal implementation of the
readFile in the Streaming API, although I am not 100% sure.
Thanks for the PR!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
