Hi, Could you elaborate on why you would like to replace the S3 client?
Best regards, Martijn On Wed, 13 Oct 2021 at 17:18, Tamir Sagi <tamir.s...@niceactimize.com> wrote: > I found the dependency > > <dependency> > <groupId>org.apache.hadoop</groupId> > <artifactId>hadoop-aws</artifactId> > <version>3.3.1</version> > </dependency> > > apparently its possible, there is a method > setAmazonS3Client > > I think I found the solution. > > Thanks. > > Tamir. > > ------------------------------ > *From:* Tamir Sagi <tamir.s...@niceactimize.com> > *Sent:* Wednesday, October 13, 2021 5:44 PM > *To:* user@flink.apache.org <user@flink.apache.org> > *Subject:* Replacing S3 Client in Hadoop plugin > > Hey community. > > I would like to know if there is any way to replace the S3 client in > Hadoop plugin[1] to a custom client(AmazonS3). > > I did notice that Hadoop plugin supports replacing the implementation of > S3AFileSystem using > "fs.s3a.impl" (in flink-conf.yaml it will be "s3.impl") but not the client > itself [2] > > <property> > <name>fs.s3a.impl</name> > <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value> > <description>The implementation class of the S3A Filesystem</description> > </property> > > I delved into Hadoop plugin source code [3] , the Client itself is of type > AmazonS3Client and cannot be replaced (for example) with a client of > type AmazonS3EncryptionV2. > > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#hadooppresto-s3-file-systems-plugins > Amazon S3 | Apache Flink > <https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#hadooppresto-s3-file-systems-plugins> > Entropy injection for S3 file systems # The bundled S3 file systems > (flink-s3-fs-presto and flink-s3-fs-hadoop) support entropy > injection.Entropy injection is a technique to improve the scalability of > AWS S3 buckets through adding some random characters near the beginning of > the key. > ci.apache.org > > [2] > https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html > <https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#S3A> > [3] > https://github.com/apache/hadoop/blob/master/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java > > Hadoop-AWS module: Integration with Amazon Web Services - Apache Hadoop > <https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#:~:text=property%3E%0A%0A%3Cproperty%3E%0A%20%20%3Cname%3E-,fs.s3a.impl,-%3C/name%3E%0A%20%20%3Cvalue%3Eorg> > Overview. Apache Hadoop’s hadoop-aws module provides support for AWS > integration. applications to easily use this support.. To include the S3A > client in Apache Hadoop’s default classpath: Make sure > thatHADOOP_OPTIONAL_TOOLS in hadoop-env.sh includes hadoop-aws in its list > of optional modules to add in the classpath.. For client side interaction, > you can declare that relevant JARs must be ... > hadoop.apache.org > Thank you, > > Best, > Tamir. > > > Confidentiality: This communication and any attachments are intended for > the above-named persons only and may be confidential and/or legally > privileged. Any opinions expressed in this communication are not > necessarily those of NICE Actimize. If this communication has come to you > in error you must take no action based on it, nor must you copy or show it > to anyone; please delete/destroy and inform the sender by e-mail > immediately. > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails. > Viruses: Although we have taken steps toward ensuring that this e-mail and > attachments are free from any virus, we advise that in keeping with good > computing practice the recipient should ensure they are actually virus free. > > > Confidentiality: This communication and any attachments are intended for > the above-named persons only and may be confidential and/or legally > privileged. Any opinions expressed in this communication are not > necessarily those of NICE Actimize. If this communication has come to you > in error you must take no action based on it, nor must you copy or show it > to anyone; please delete/destroy and inform the sender by e-mail > immediately. > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails. > Viruses: Although we have taken steps toward ensuring that this e-mail and > attachments are free from any virus, we advise that in keeping with good > computing practice the recipient should ensure they are actually virus free. >
