Re: Long running Flink job on yarn is getting failed after some time.

Hi, Sorry for previous mail it was sent by mistake. Flink version: 1.3.2 EMR with latest yarn. java.lang.RuntimeException: Exception occurred while processing valve output watermark: at org.apache.flink.streaming.runtime.io.StreamInputProcessor$ForwardingValveOutputHandler.handleWatermark(Stream

Re: Long running Flink job on yarn is getting failed after some time.

Can tou eleborate more please. On 13 Dec 2017 9:24, "Shivam Sharma" <28shivamsha...@gmail.com> wrote: > Hi, > > Flink version: 1.3.2 > > -- > Shivam Sharma > Data Engineer @ Goibibo > Indian Institute Of Information Technology, Design and Manufacturing > Jabalpur > Mobile No- (+91) 8882114744 >

Long running Flink job on yarn is getting failed after some time.

Hi, Flink version: 1.3.2 -- Shivam Sharma Data Engineer @ Goibibo Indian Institute Of Information Technology, Design and Manufacturing Jabalpur Mobile No- (+91) 8882114744 Email:- 28shivamsha...@gmail.com LinkedIn:-*https://www.linkedin.com/in/28shivamsharma

[jira] [Created] (FLINK-8247) Support Hadoop-free variant of Flink on Mesos

Eron Wright created FLINK-8247: --- Summary: Support Hadoop-free variant of Flink on Mesos Key: FLINK-8247 URL: https://issues.apache.org/jira/browse/FLINK-8247 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-8246) Download page must not link to dist.apache.org

Sebb created FLINK-8246: --- Summary: Download page must not link to dist.apache.org Key: FLINK-8246 URL: https://issues.apache.org/jira/browse/FLINK-8246 Project: Flink Issue Type: Bug Environme

[jira] [Created] (FLINK-8245) Add some interval between logs when requested resources are not available

Wind created FLINK-8245: --- Summary: Add some interval between logs when requested resources are not available Key: FLINK-8245 URL: https://issues.apache.org/jira/browse/FLINK-8245 Project: Flink Issue

Re: Storing large lists into state per key

Hi Jan, You could associate a key to each element of your Key's list (e.g., hashing the value), keep only the keys in heap (e.g., in a list) and the associated state key-value/s in an external store like RocksDB/Redis, but you will notice large overheads due to de/serializing - a huge penatly f

[jira] [Created] (FLINK-8244) There are two zookeeper client created when read data from hbase in the flink yarn session model

zhanglibing000 created FLINK-8244: - Summary: There are two zookeeper client created when read data from hbase in the flink yarn session model Key: FLINK-8244 URL: https://issues.apache.org/jira/browse/FLINK-8244

Re: [ANNOUNCE] Apache Flink 1.4.0 released

Thank you Aljoscha for managing the release! 2017-12-12 12:46 GMT+01:00 Aljoscha Krettek : > The Apache Flink community is very happy to announce the release of Apache > Flink 1.4.0. > > Apache Flink® is an open-source stream processing framework for > distributed, high-performing, always-availab

[ANNOUNCE] Apache Flink 1.4.0 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.4.0. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available for download at: https:

[jira] [Created] (FLINK-8243) OrcTableSource should recursively read all files in nested directories of the input path.

Fabian Hueske created FLINK-8243: Summary: OrcTableSource should recursively read all files in nested directories of the input path. Key: FLINK-8243 URL: https://issues.apache.org/jira/browse/FLINK-8243

[jira] [Created] (FLINK-8242) ClassCastException in OrcTableSource.toOrcPredicate

Fabian Hueske created FLINK-8242: Summary: ClassCastException in OrcTableSource.toOrcPredicate Key: FLINK-8242 URL: https://issues.apache.org/jira/browse/FLINK-8242 Project: Flink Issue Type:

Re: Storing large lists into state per key

Hi Fabian, thanks for quick reply, what you suggest seems to work at first sight, I will try it. Is there any reason not to implement a RocksDBListState this way in general? Is there any increased overhead of this approach? Thanks,  Jan On 12/12/2017 11:17 AM, Fabian Hueske wrote: Hi Jan,

Re: Storing large lists into state per key

Hi Jan, I cannot comment on the internal design, but you could put the data into a RocksDBStateBackend MapState where the value X is your data type and the key is the list index. You would need another ValueState for the current number of elements that you put into the MapState. A MapState allows

Re: [RESULT][VOTE] Release 1.4.0, release candidate #3

Ah, you're right, the subject has it right, though. > On 12. Dec 2017, at 11:06, Alexey Demin wrote: > > Hi > >>> The voting time has passed and I'm happy to announce that we've > collected enough votes to release this RC as Flink 1.3.2. > > Maybe 1.4.0, because 1.3.2 already exist =) > > >

Re: [RESULT][VOTE] Release 1.4.0, release candidate #3

Hi >> The voting time has passed and I'm happy to announce that we've collected enough votes to release this RC as Flink 1.3.2. Maybe 1.4.0, because 1.3.2 already exist =) 2017-12-12 13:58 GMT+04:00 Aljoscha Krettek : > The voting time has passed and I'm happy to announce that we've collected

[RESULT][VOTE] Release 1.4.0, release candidate #3

The voting time has passed and I'm happy to announce that we've collected enough votes to release this RC as Flink 1.3.2. +1 votes: - Piotrek (non-binding) - Stefan (non-binding) - Fabian (binding) - Chesnay (binding) - Timo (binding) - Tzu-Li (binding) - Aljoscha (binding) That's 7 votes, 5 bi

Re: [VOTE] Release 1.4.0, release candidate #3

+1 Verified: - NOTICE and LICENSE are correct - source doesn't contain binaries - verified signatures - verified hashes - cluster testing on AWS and Cloudera VM (with Kerberos) (see release-testing doc) - verified "mvn clean verify" for all supported Hadoop versions (2.4.1 to 2.9.0) > On 11. Dec

Storing large lists into state per key

Hi all, I have a question that appears as a user@ question, but brought me into the dev@ mailing list while I was browsing through the Flink's source codes. First I'll try to briefly describe my use case. I'm trying to do a group-by-key operation with a limited number of distinct keys (which I