Hi Nick, What version of Hadoop are you using? AFAIK, you must use Hadoop 2.7+ to support custom s3 endpoint, or the `fs.s3a.endpoint` property in core-site.xml would be ignored.
Best, Paul Lam > 在 2018年12月19日，06:40，Martin, Nick <nick.mar...@ngc.com> 写道： > > I’m working on Flink 1.7.0 and I’m trying to use the built in S3 libraries > like readFile(‘s3://bucket/object’ <s3://bucket/object%E2%80%99>) or > StreamingFileSink. My storage provider is not AWS, but they implement the > same API. So I need to point the S3 client to a different address. The Hadoop > documentation shows that there are options in core-site to set that up. The > problem is, I can’t seem to get the right dependencies together to use the S3 > filesystem. As far as I can tell, the pre-built Hadoop/Presto jars don’t use > core-site.xml, and the instructions for manual setup given here > (https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/deployment/aws.html#hadoop-provided-s3-file-systems---manual-setup > > <https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/deployment/aws.html#hadoop-provided-s3-file-systems---manual-setup>) > list a set of dependencies that seems to be completely wrong. > > How can use the S3 sources/sinks with a custom http endpoint? > > > > Nick Martin > > > Notice: This e-mail is intended solely for use of the individual or entity to > which it is addressed and may contain information that is proprietary, > privileged and/or exempt from disclosure under applicable law. If the reader > is not the intended recipient or agent responsible for delivering the message > to the intended recipient, you are hereby notified that any dissemination, > distribution or copying of this communication is strictly prohibited. This > communication may also contain data subject to U.S. export laws. If so, data > subject to the International Traffic in Arms Regulation cannot be > disseminated, distributed, transferred, or copied, whether incorporated or in > its original form, to foreign nationals residing in the U.S. or abroad, > absent the express prior approval of the U.S. Department of State. Data > subject to the Export Administration Act may not be disseminated, > distributed, transferred or copied contrary to U. S. Department of Commerce > regulations. If you have received this communication in error, please notify > the sender by reply e-mail and destroy the e-mail message and any physical > copies made of the communication. > Thank you. > *********************
