Hi Amogh
You can test the code below:
```java
FileSource.forRecordStreamFormat(csvFormat, new Path("file:///tmp/test"))
.setFileEnumerator(() -> new NonSplittingRecursiveEnumerator(path ->
path.getName().endsWith(".csv")))
.build();
```
Best,
Feng
On Sat, Aug 17, 2024 at 4:41 AM amogh joshi <amoghjo...@gmail.com> wrote:
> Hi Users,
>
> Any clues on configurable regex path for FilesSource/Filesystem connector
> for stream APIs is appreciated.
>
> Regards,
> Amogh.
>
>
> On Thu, 15 Aug, 2024, 11:18 amogh joshi, <amoghjo...@gmail.com> wrote:
>
>> Hi,
>>
>> I am building a pretty straightforward processing pipeline as described
>> below, using *DataStream* *APIs* and *FileSystem connector*.
>>
>> *filesystem-source -> transforms -> database-sink*
>>
>> Everything worked well till the filesystem (source) had just a single
>> type (JSON) of files. Recently the filesystem got modified to store other
>> types of files too (like parquet, text, etc.).
>>
>> It appeared just a matter of specifying some regex to the FileSource or
>> SplitEnumerator and creating different sources for different types of
>> files.
>>
>> However, the two public methods *FileSource::forRecordStreamFormat *and
>> *FileSource::forBulkFileFormat* do not allow configuring the regex based
>> file paths.
>>
>> What could be the recommended way to configure a custom regex for a
>> FileSource?
>>
>> I am using version 1.19.1.
>>
>> Regards,
>> Amogh.
>>
>
