Re: S3 Source support in Flink

Hi, One more additional link, the overall Sources documentation page also contains useful information. [1] Best regards, Martijn [1] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/sources/ On Fri, 29 Oct 2021 at 09:21, Martijn Visser wrote: > Hi, > > When usin

Re: S3 Source support in Flink

Hi, When using the DataStream API, the new File Source already supports continuous stream, but it isn't documented yet [1] There is an e2e test for it, so you could look at that to understand how it works. It's not yet supported for the Table API/SQL [2] Best regards, Martijn [1] https://issues

Re: S3 Source support in Flink

Hi Abhishek, You can use `readFileStream` directly defined on DataStream. You will still have to pay the ListObjects for each iteration using that method. If you want a source that does not rely on listing, you can implement a custom SQS source (there is no official existing one currently) and use

S3 Source support in Flink

Hello, I see S3 supported as a Sink through StreamingFileSink but do not see a source equivalent StreamingFileSource *Questions:* 1. What is the current recommendation for using S3 as a continuous