Re: Flink File Source: File read strategy

hat File Source reads files directly from S3? > > > > Regards, > > Kirti Dhar > > > > *From:* Shammon FY > *Sent:* 25 September 2023 06:27 > *To:* Kirti Dhar Upadhyay K > *Cc:* user@flink.apache.org > *Subject:* Re: Flink File Source: File read strategy &

RE: Flink File Source: File read strategy

Thanks Shammon. Is there any way to verify that File Source reads files directly from S3? Regards, Kirti Dhar From: Shammon FY Sent: 25 September 2023 06:27 To: Kirti Dhar Upadhyay K Cc: user@flink.apache.org Subject: Re: Flink File Source: File read strategy Hi Kirti, I think the default

Re: Flink File Source: File read strategy

Hi Kirti, I think the default file `Source` does not download files locally in Flink, but reads them directly from S3. However, Flink also supports configuring temporary directories through `io.tmp.dirs`. If it is a user-defined source, it can be obtained from FlinkS3FileSystem. After the Flink jo

Flink File Source: File read strategy

Hi Community, I am using Flink File Source with Amazon S3. Please help me on below questions- 1. When Split Enumerator assigns split to Source Reader, does it downloads the file temporarily and then starts reading/decoding the records from file or it creates direct stream with S3? 1. I