Re: Questions on S3 File Sink Behavior

2023-03-29 Thread Mate Czagany
Hi, 1. In case of S3 FileSystem, Flink uses the multipart upload process [1] for better performance. It might not be obvious at first by looking at the docs, but it's noted at the bottom of the FileSystem page [2] For more information you can also check FLINK-9751 and FLINK-9752 2. In case of loc

Questions on S3 File Sink Behavior

2023-03-29 Thread Chirag Dewan via user
Hi,   We are tying to use Flink's File sink to distribute files to AWS S3 storage. We are using Flink provided Hadoop s3a connector as plugin. We have some observations that we needed to clarify: 1. When using file sink for local filesystem distribution, we can see that the sink creates 3 se