Glad it worked.
At the same time, I think adding filters to the FileSystem Source is a
fairly common scenario, but the current documentation lacks relevant
examples.
We could improve this section with examples and also add relevant methods
to the Builder class to facilitate passing the necessary f
Thanks Feng.
Above code worked well and served its purpose.
I just modified it to use *NonSplittingRecursiveAllDirEnumerator* instead
of *NonSplittingRecursiveEnumerato**r* as the regex filter was to be
applied on the individual filenames, excluding the parent directory
specified as the path.
Fi
Hi Amogh
You can test the code below:
```java
FileSource.forRecordStreamFormat(csvFormat, new Path("file:///tmp/test"))
.setFileEnumerator(() -> new NonSplittingRecursiveEnumerator(path ->
path.getName().endsWith(".csv")))
.build();
```
Best,
Feng
On Sat, Aug 17, 2024 at 4:41 AM amogh joshi
Hi Users,
Any clues on configurable regex path for FilesSource/Filesystem connector
for stream APIs is appreciated.
Regards,
Amogh.
On Thu, 15 Aug, 2024, 11:18 amogh joshi, wrote:
> Hi,
>
> I am building a pretty straightforward processing pipeline as described
> below, using *DataStream* *AP
Hi,
I am building a pretty straightforward processing pipeline as described
below, using *DataStream* *APIs* and *FileSystem connector*.
*filesystem-source -> transforms -> database-sink*
Everything worked well till the filesystem (source) had just a single type
(JSON) of files. Recently the fil
