Hi, sorry, yes, you don't have to add any of the Hadoop dependencies. Everything that's needed comes in the presto s3 jar.
You should use "s3:" as the prefix, the Presto S3 filesystem will not be used if you use s3a. And yes, you add config values to the flink config as s3.xxx. Best, Aljoscha > On 5. Jun 2018, at 18:23, Hao Sun <ha...@zendesk.com> wrote: > > Thanks for pick up my question. I had s3a in the config now I removed it. > I will post a full trace soon, but want to get some questions answered to > help me understand this better. > > 1. Can I use the presto lib with Flink 1.5 without bundled hdp? Can I use > this? > http://www.apache.org/dyn/closer.lua/flink/flink-1.5.0/flink-1.5.0-bin-scala_2.11.tgz > > <http://www.apache.org/dyn/closer.lua/flink/flink-1.5.0/flink-1.5.0-bin-scala_2.11.tgz> > 2. How do I configure presto for endpoints, encryption? The S3A file system > needed core-site.yml to configure such things and S3 V4 signature. Do I have > to do it for presto? > 3. If yes, how to do it? Just add s3.xxx to flink-config? > like s3.server-side-encryption-algorithm: AES256 > s3.endpoint: 's3.amazonaws.com <http://s3.amazonaws.com/>' other values > for France regions, etc > > I will post more logs when I get one. Thanks > > On Tue, Jun 5, 2018 at 9:09 AM Aljoscha Krettek <aljos...@apache.org > <mailto:aljos...@apache.org>> wrote: > Hi, > > what are you using as the FileSystem scheme? s3 or s3a? > > Also, could you also post the full stack trace, please? > > Best, > Aljoscha > > >> On 2. Jun 2018, at 07:34, Hao Sun <ha...@zendesk.com >> <mailto:ha...@zendesk.com>> wrote: >> >> I am trying to figure out how to use S3 as state storage. >> The recommended way is >> https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/aws.html#shaded-hadooppresto-s3-file-systems-recommended >> >> <https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/aws.html#shaded-hadooppresto-s3-file-systems-recommended> >> >> Seems like I only have to do two things: >> 1. Put flink-s3-fs-presto to the lib >> 2. Configure >> s3.access-key: your-access-key >> s3.secret-key: your-secret-key >> >> But I see this exception: ClassNotFoundException: >> NativeS3FileSystem/S3AFileSystem Not Found >> >> https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/aws.html#provide-s3-filesystem-dependency >> >> <https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/aws.html#provide-s3-filesystem-dependency> >> >> Add it is suggested to add more libs. >> So I am confused here, is there a step 3 needed? Isn't the presto jar is all >> self contained? >> >> Thanks >
