Hi Users, Any clues on configurable regex path for FilesSource/Filesystem connector for stream APIs is appreciated.
Regards, Amogh. On Thu, 15 Aug, 2024, 11:18 amogh joshi, <amoghjo...@gmail.com> wrote: > Hi, > > I am building a pretty straightforward processing pipeline as described > below, using *DataStream* *APIs* and *FileSystem connector*. > > *filesystem-source -> transforms -> database-sink* > > Everything worked well till the filesystem (source) had just a single type > (JSON) of files. Recently the filesystem got modified to store other types > of files too (like parquet, text, etc.). > > It appeared just a matter of specifying some regex to the FileSource or > SplitEnumerator and creating different sources for different types of > files. > > However, the two public methods *FileSource::forRecordStreamFormat *and > *FileSource::forBulkFileFormat* do not allow configuring the regex based > file paths. > > What could be the recommended way to configure a custom regex for a > FileSource? > > I am using version 1.19.1. > > Regards, > Amogh. >
