Re: Documentation for translation of Job graph to Execution graph

Hi, Thanks for sharing this link. I have not see it before. May be this is newly added in 1.0 docs. I will go through it. In general, there are two things I am trying to understand and get comfortable with - 1. How a Job graph is translated to Execution graph. The logs and monitoring APIs

Re: Kinesis connector classpath issue when running Flink 1.1-SNAPSHOT on YARN

Hi Aljoscha, Thanks! It looks like you're right. I've ran it with the FsStateBackend and everything works fine. I've also got it working with RocksDBStateBackend now, by rebuilding Flink master with: - the verify step in FsStateBackend skipped for URIs with s3 schemes. - the initialisation of fil

Re: Kinesis connector classpath issue when running Flink 1.1-SNAPSHOT on YARN

Hi, I think the problem with the missing Class com.amazon.ws.emr.hadoop.fs.EmrFileSystem is not specific to RocksDB. The exception is thrown in the FsStateBackend, which is internally used by the RocksDB backend to do snapshotting of non-partitioned state. The problem is that the FsStateBackend tri

Re: Kinesis connector classpath issue when running Flink 1.1-SNAPSHOT on YARN

I found that I can still write to s3, using my Flink build of 1.1-SNAPSHOT, for example if I run the word count example: ./bin/flink run ./examples/batch/WordCount.jar --input hdfs:///tmp/LICENSE --output s3://permutive-flink/wordcount-result.txt This works fine - it's just the RocksDBStateBackend

Flink on YARN : Amazon S3 wrongly used instead of HDFS

Hi everyone, I followed Flink on YARN's setup documentation. But when I run with ./bin/yarn-session.sh -n 2 -jm 1024 -tm 2048, while being authenticated to Kerberos, I get the following error : 2016-06-16 17:46:

Re: Kinesis connector classpath issue when running Flink 1.1-SNAPSHOT on YARN

Hi Gordon/Fabian, Thanks for helping with this! Downgrading the Maven version I was using to build Flink appears to have fixed that problem - I was using Maven 3.3.3 before and have downgraded to 3.2.5. Just for reference, I printed the loaded class at runtime and found that when I was using Flin

Re: Documentation for translation of Job graph to Execution graph

Hi, I'm afraid there is no documentation besides the link that you posted and this one: https://ci.apache.org/projects/flink/flink-docs-release-1.0/concepts/concepts.html . With what parts are you having trouble? Maybe I can help. Cheers, Aljoscha On Thu, 16 Jun 2016 at 19:31 Bajaj, Abhinav wro

Re: Kinesis connector classpath issue when running Flink 1.1-SNAPSHOT on YARN

Hi Josh, I assume that you build the SNAPSHOT version yourself. I had similar version conflicts for Apache HttpCore with Flink SNAPSHOT versions on EMR. The problem is cause by a changed behavior in Maven 3.3 and following versions. Due to these changes, the dependency shading is not working corre