Hi,
We are tying to use Flink's File sink to distribute files to AWS S3 storage. We are using Flink provided Hadoop s3a connector as plugin. We have some observations that we needed to clarify: 1. When using file sink for local filesystem distribution, we can see that the sink creates 3 sets of files - in progress, pending (on rolling) and finished (upon checkpointing). But with S3 file sink we can see only the finished files, in the S3 buckets. So we wanted to understand where does the sink creates the in-progress and pending files for S3 file sink ? 2. We can also see with local file system sink, the in-progress and pending file names follow the nomenclature:.<prefix>-<uid>-<partFileIndex>.inprogress.uid-<suffix> There is a dot at the begining of the filename, may be flink is trying to create these files as hidden files. But in the flink documentation this is not mentioned. So can we assume that the in-progress and pending filenames shall always start with a dot ? thanks a lot in advance
