Re: Using S3 as state backend

2015-12-16 Thread Thomas Götzinger
Hi Brian, thanks, that helped me a lot. 2015-12-15 16:52 GMT+01:00 Brian Chhun : > Sure, excuse me if anything was obvious or wrong, I know next to nothing > about Hadoop. > > 1. get the Hadoop 2.7 distribution (I set its path to HADOOP_HOME to make > things easier for mysellf) > 2. set the HA

Re: Using S3 as state backend

2015-12-15 Thread Brian Chhun
Sure, excuse me if anything was obvious or wrong, I know next to nothing about Hadoop. 1. get the Hadoop 2.7 distribution (I set its path to HADOOP_HOME to make things easier for mysellf) 2. set the HADOOP_CLASSPATH to include ${HADOOP_HOME}/share/hadoop/common/*:${HADOOP_HOME}/share/hadoop/tools/

Re: Using S3 as state backend

2015-12-14 Thread Thomas Götzinger
Hi Brian Can you give me short summary how to achieve this. Am 14.12.2015 23:20 schrieb "Brian Chhun" : > For anyone else looking, I was able to use the s3a filesystem which can > use IAM role based authentication as provided by the underlying AWS client > library. > > Thanks, > Brian > > On Thu,

Re: Using S3 as state backend

2015-12-14 Thread Brian Chhun
For anyone else looking, I was able to use the s3a filesystem which can use IAM role based authentication as provided by the underlying AWS client library. Thanks, Brian On Thu, Dec 10, 2015 at 4:28 PM, Brian Chhun wrote: > Thanks Ufuk, this did the trick. > > Thanks, > Brian > > On Wed, Dec 9,

Re: Using S3 as state backend

2015-12-10 Thread Brian Chhun
Thanks Ufuk, this did the trick. Thanks, Brian On Wed, Dec 9, 2015 at 4:37 PM, Ufuk Celebi wrote: > Hey Brian, > > did you follow the S3 setup guide? > https://ci.apache.org/projects/flink/flink-docs-master/apis/example_connectors.html > > You have to set the fs.hdfs.hadoopconf property and add

Re: Using S3 as state backend

2015-12-09 Thread Ufuk Celebi
Hey Brian, did you follow the S3 setup guide? https://ci.apache.org/projects/flink/flink-docs-master/apis/example_connectors.html You have to set the fs.hdfs.hadoopconf property and add fs.s3.impl org.apache.hadoop.fs.s3native.NativeS3FileSystem to core-site.xml – Ufuk > On 09 Dec 2015, at

Using S3 as state backend

2015-12-09 Thread Brian Chhun
Hello, I'm trying to setup an HA cluster and I'm running into issues using S3 as the state backend. This is raised during startup: 2015-12-09T19:23:36.430724+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: java.io.IOException: No file system found with scheme s3, referenced in file URI 's3://