Yaroslav Tkachenko created FLINK-30873:
------------------------------------------

             Summary: FileSystem Table API connector doesn't discover new files 
when partitions are specified
                 Key: FLINK-30873
                 URL: https://issues.apache.org/jira/browse/FLINK-30873
             Project: Flink
          Issue Type: Bug
          Components: Connectors / FileSystem
    Affects Versions: 1.16.1
            Reporter: Yaroslav Tkachenko


When *source.monitor-interval* is configured I expect the source connector to 
discover new files. However, if the source table was created with partitions 
(using PARTITIONED BY), it only discovers new files in the partition folders 
that existed during the startup; it doesn't discover new partitions. 

I believe the problem is 
[here|https://github.com/apache/flink/blob/5f2d088a2713ced5c6ce072db92f4378f73bc739/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/FileSystemTableSource.java#L276-L286],
 only these file paths are monitored internally, no partition discovery is 
currently implemented. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to