Hi Marco, ideally you solve everything with IAM roles, but you can also use credentials providers such as EnvironmentVariableCredentialsProvider[1].
The key should be s3.aws.credentials.provider: com.amazonaws.auth.EnvironmentVariableCredentialsProvider Remember to put the respective jar into the folder of your s3p plugin, the folder structure should look like described here[2]. Note that this is tested for s3a, so it could be that it works differently in s3p. I see that presto usually uses presto.s3.credentials-provider. [1] https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/EnvironmentVariableCredentialsProvider.html [2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/filesystems/plugins.html On Thu, Jan 28, 2021 at 7:00 PM Marco Villalobos <[email protected]> wrote: > Is it possible to use an environmental credentials provider? > > On Thu, Jan 28, 2021 at 8:35 AM Arvid Heise <[email protected]> wrote: > >> Hi Marco, >> >> afaik you don't need HADOOP_HOME or core-site.xml. >> >> I'm also not sure from where you got your config keys. (I guess from the >> Presto page, which probably all work if you remove hive., maybe we should >> also support that) >> >> All keys with prefix s3 or s3p (and fs.s3, fs.s3p) are routed towards >> presto [1]. >> >> So it should be >> s3.access-key: XXX >> s3.secret-key: XXX >> s3.endpoint: http://aws:4566 >> s3.path-style-access: true >> s3.path.style.access: true (only one of them is needed, but I don't know >> which, so please try out) >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-stable/deployment/filesystems/s3.html#configure-access-credentials >> >> On Thu, Jan 28, 2021 at 4:58 PM Marco Villalobos < >> [email protected]> wrote: >> >>> Hi, >>> >>> I got s3a working on localstack. The missing piece of information from >>> Flink documentation seems to be that the system requires a HADOOP_HOME and >>> core-site.xml. >>> >>> Flink documentation states that s3p (presto) should be used for file >>> checkpointing into s3. I am using RocksDB, which I assume also means that I >>> should use s3p (the documentation was not specific about that). Is that >>> assumption correct? >>> >>> However, I cannot get s3p working now. >>> >>> I did the following so far: >>> >>> I created the s3-fs-presto plugin directory and copied the jar from the >>> opt directory there. >>> I am not sure where to put the configuration keys though. The >>> documentation states that I can just put in my flink-conf.yaml, but I had >>> no success. >>> >>> Where do I put the presto configuration keys? Are there any other >>> missing steps? Is this something that would only work on an EMR environment >>> with a real HIVE running? >>> >>> # The S3 storage endpoint server. This can be used to connect to an >>> S3-compatible storage >>> # system instead of AWS. When using v4 signatures, it is recommended to >>> set this to the >>> # AWS region-specific endpoint (e.g., http[s]://<bucket>.s3-<AWS-region>. >>> amazonaws.com). >>> hive.s3.endpoint: http://aws:4566 >>> >>> # Use HTTPS to communicate with the S3 API (defaults to true). >>> hive.s3.ssl.enabled: false >>> >>> # Use path-style access for all requests to the S3-compatible storage. >>> # This is for S3-compatible storage that doesn’t support >>> virtual-hosted-style access. (defaults to false) >>> hive.s3.path-style-access: true >>> >>> But that also did not work. >>> >>> Any advice would be appreciated. >>> >>> -Marco Villalobos >>> >>
