merge/union fast and slow streams based on event timestamp

2018-04-30 Thread Tao Xia
I am running into a problem when processing the past 7 days of data from multiple streams. I am trying to union the streams based on event timestamp. The problem is that there are streams are significant big than other streams. For example if one stream has 1,000 event/sec and the other stream ha

RE: Fat jar fails deployment (streaming job too large)

2018-04-30 Thread Chan, Regina
Any updates on this one? I'm seeing similar issues with 1.3.3 and the batch api. Main difference is that I have even more operators ~850, mostly maps and filters with one cogroup. I don't really want to set a akka.client.timeout for anything more than 10 minutes seeing that it still fails with

Best way to clean-up states in memory

2018-04-30 Thread ashish pok
All, I am using noticing heap utilization creeping up slowly in couple of apps which eventually lead to OOM issue. Apps only have 1 process function that cache state. I did make sure I have a clear method invoked when events are collected normally, on exception and on timeout. Are any other best

Re: Class loading issues when using Remote Execution Environment

2018-04-30 Thread kedar mhaswade
Chesnay, I have filed https://issues.apache.org/jira/browse/FLINK-9267 to keep track of this issue. Regards, Kedar On Fri, Apr 27, 2018 at 11:50 AM, kedar mhaswade wrote: > Thanks again! > > This is strange. With both Flink 1.3.3 and Flink 1.6.0-SNAPSHOT and > 1) copying gradoop-demo-shaded.ja

Apache Flink - Flink Forward SF 2018 - Scaling stream data pipelines (source code)

2018-04-30 Thread M Singh
Hi: I was looking at the flink-forward sf 2018 presentations and wanted to find out if there is a git repo where I can find the code for "Scaling stream data pipelines" by Till Rohrmann & Flavio Junqueira ? Thanks

RE: use of values of previously accepted event

2018-04-30 Thread Esa Heikkinen
Hi I am still interested in getting an answer, if anyone can help ? If I have a pattern sequence like this: val eventPattern = Pattern .begin[TestData]("start").where( // Sets variable X here //) .followedBy("end").where( // Reads value of variable X here //) How to set variable X in "sta

Re: Raspberry Pi Memory Configuration

2018-04-30 Thread Ankit Chaudhary
Hey Nicholas, I am not sure if the problem of Flink dropping TM is connected to ethernet because it happens in our Prod cluster too where we have standard blade servers with Gigabyte network (din spent time yet on doing the RCA). Nonetheless, its really cool what you are doing there with rPis :))