Hi Jark, Thank you for your helpful and quick response, I will try the hive connector solution. It will be great if you can point out the link tracking the filesystem streaming data source.
On Mon, Nov 23, 2020 at 10:33 AM Jark Wu <imj...@gmail.com> wrote: > Hi Kai, > > Streaming filesystem source is not supported yet in TableAPI/SQL. > This is on the roadmap and there are some problems that need to be fixed. > As a workaround, you can use Hive connector to reading files continuously > on > filesystems [1]. > > Best, > Jark > > [1]: > https://ci.apache.org/projects/flink/flink-docs-master/dev/table/hive/hive_read_write.html#streaming-source-enable > > > > > > On Mon, 23 Nov 2020 at 10:21, Xingbo Huang <hxbks...@gmail.com> wrote: > >> Hi Kai, >> >> I took a look at the implementation of the filesystem connector. It will >> decide which files to read at startup >> and won't change during running. If you want to need this function, you >> may need to customize a new connector. >> >> Best, >> Xingbo >> >> eef hhj <zzfu...@gmail.com> 于2020年11月21日周六 下午2:38写道： >> >>> Hi, >>> >>> I'm facing a situation where I want the Flink App to dynamically detect >>> the change of the Filesystem batch data source. As I tried in the following >>> example in sql-client.sh, it can query all the records under the folder for >>> the select. >>> >>> While I'm adding a new file to the folder, the query does not refresh >>> and it seems it cannot detect the new file. It can only reflect records in >>> the new file unless I cancel the current query and do select again. Is it >>> possible to make the App detect such file changes automatically as the one >>> in the stream source? >>> >>> CREATE TABLE fs_table ( >>> user_id STRING, >>> order_amount DOUBLE, >>> dt STRING, >>> hh STRING, >>> ) WITH ( >>> 'connector'='filesystem', >>> 'path'='file:///path/folder/', >>> 'format'='csv' >>> ); >>> >>> select * from fs_table; >>> >>> -- Best wishes >>> Kai >>> >> -- Best regards, Fu Kai（扶凯）
