Reference configs for HA / RocksDB / YARN / Zookeeper / HDFS

2017-03-10 Thread Torok, David
Hi, Forgive me if parts of this question have been answered before but I'd like help in resolving some bits of confusion from the documentation and the fact that I haven't been able to find a good example anywhere for an enterprise-style setup. If anyone has a sample HA / Yarn / ZK / RocksDB

Reference configs for HA / RocksDB / YARN / Zookeeper / HDFS

2017-03-10 Thread Torok, David
Hi, Forgive me if parts of this question have been answered before but I'd like help in resolving some bits of confusion from the documentation and the fact that I haven't been able to find a good example anywhere for an enterprise-style setup. If anyone has a sample HA / Yarn / ZK / RocksDB

Amazon EMR Releases

2017-03-06 Thread Torok, David
Does anyone have any insight as to how closely Amazon EMR releases will track the Flink releases? For example EMR 5.3.1 on Feb 7 included Flink 1.1.4, from Dec 21 ... about 1.5 month lag. With Flink accelerating to a timed release schedule, I wonder how far behind EMR will track the official

Re: RocksDB Windows / Flink 1.1.4 - requires Hadoop?

2016-12-23 Thread Torok, David
Thanks for the prompt reply Stephan It is working perfectly now! Best Regards, Dave >Hi Dave! > >In 1.1.x, the default mode of RocksDB uses some Hadoop utilities, regardless >of the filesystem used. I think that was a design mistake and we rectified >that in the upcoming 1.2.x > >For 1.1.4, I

RocksDB Windows / Flink 1.1.4 - requires Hadoop?

2016-12-23 Thread Torok, David
Hi, I tried to use the newly-supported RocksDB backend (Flink 1.1.4) on my Windows laptop. However, it is not creating any state and is throwing NPE while trying to call: org.apache.flink.streaming.util.HDFSCopyFromLocal$1.run(HDFSCopyFromLocal.java:47) which eventually gets down to no

Update avro to 1.7.7 or later for flink 1.1.4

2016-12-04 Thread Torok, David
I spent close to two days and tracked down the solution to a major issue with Avro / GenericRecord and Flink. In short, there is a field marked 'transient' in Avro 1.7.6 and earlier which interferes with correct Kryo serialization. This was fixed in Avro 1.7.7, but Flink is dependent on Avro 1