If you are using both the Hadoop S3 and Presto S3 filesystems, you should use s3p:// and s3a:// to distinguish between the two.
Presto is recommended for checkpointing because the Hadoop implementation has very high latency when creating files, and because it hits request rate limits very quickly. The Hadoop S3 filesystem tries to imitate a normal filesystem on top of S3: - before writing a key it checks if the "parent directory" exists by checking for a key with the prefix up to the last "/" - it creates empty marker files to mark the existence of such a parent directory - all these existence requests are S3 HEAD requests, which have very low request rate limits *David Anderson* | Training Coordinator Follow us @VervericaData -- Join Flink Forward - The Apache Flink Conference Stream Processing | Event Driven | Real Time On Fri, Mar 27, 2020 at 7:10 AM <b.z...@dell.com> wrote: > Hi, > > > > In this document > https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/s3.html#hadooppresto-s3-file-systems-plugins, > it mentioned that > > - Presto is the recommended file system for checkpointing to S3. > > Is there a reason for that? Is there some bottleneck for s3 hadoop plugin > that can’t support checkpoint storage well? > > > > And if I have the s3:// scheme with both plugins loaded, is there a class > loading order or just random for accessing S3? Which plugin will take > charge? > > > > Best Regards, > > Brian > > >
