Flink memory usage

Hi, I would like to share some considerations about Flink memory consumption. I have a cluster composed of three nodes: 1 used both as JM and TM and other 2 TM. I ran two identical applications (in different moments) on it. The only difference is that on the second one I doubled every operators, e

Re: FlinkCEP, circular references and checkpointing failures

Thank you very much for your steady response, Kostas! Cheers, Federico 2017-11-03 16:26 GMT+01:00 Kostas Kloudas : > Hi Federico, > > Thanks for trying it out! > Great to hear that your problem was fixed! > > The feature freeze for the release is going to be next week, and I would > expect 1 or

Re: Initialise side input state

Hi Maxim, thanks for the explanation. I think you can set a ValueState and a ListState for the price and the purchase events, separately. On one hand, when receiving a purchase event, you first check the price state. If it exists, you just collect the PurchaseTotal result; otherwise you can tempor

Re: Making external calls from a FlinkKafkaPartitioner

Thanks Nico - Thanks for the feedback, and nice catch on the missing volatile. Ron — Ron Crocker Principal Engineer & Architect ( ( •)) New Relic rcroc...@newrelic.com M: +1 630 363 8835 > On Nov 3, 2017, at 7:48 AM, Nico Kruber wrote: > > Hi Ron, > imho your code should be fine (except for a

Re: FlinkCEP, circular references and checkpointing failures

Hi Federico, Thanks for trying it out! Great to hear that your problem was fixed! The feature freeze for the release is going to be next week, and I would expect 1 or 2 more weeks testing. So I would say in 2.5 weeks. But this is of course subject to potential issues we may find during testing

Re: FlinkCEP, circular references and checkpointing failures

Hi Kostas, I just tried running the same job with 1.4-SNAPSHOT for 10 minutes and it didn't crash, so that was the same underlying issue of the JIRA you linked. Do you happen to know when it's expected the 1.4 stable release? Thank you very much, Federico 2017-11-03 15:25 GMT+01:00 Kostas Kloud

Re: Using Flink Ml with DataStream

Hi Chesnay, Thanks for the reply, do you know how to serve using the trained model? Where is the model saved? Regards, Adarsh ‌ On Wed, Nov 1, 2017 at 4:46 PM, Chesnay Schepler wrote: > I don't believe this to be possible. The ML library works exclusively with > the Batch API. > > > On 30.

Re: Incremental checkpointing documentation

Hi Elias, let me answer the questions to the best of my knowledge, but in general I think this is as expected. (Let me give a link to the docs explaining the activation [1] for other readers first.) On Friday, 3 November 2017 01:11:52 CET Elias Levy wrote: > What is the interaction of incrementa

Re: Making external calls from a FlinkKafkaPartitioner

Hi Ron, imho your code should be fine (except for a potential visibility problem on the changes of the non-volatile partitionMap member, depending on your needs). The #open() method should be called (once) for each sink initialization (according to the javadoc) and then you should be fine with t

Re: FlinkCEP, circular references and checkpointing failures

Perfect! thanks a lot! Kostas > On Nov 3, 2017, at 3:23 PM, Federico D'Ambrosio > wrote: > > Hi Kostas, > > yes, I'm using 1.3.2. I'll try the current master and I'll get back to you. > > 2017-11-03 15:21 GMT+01:00 Kostas Kloudas >: > Hi Federico, > > I

Re: FlinkCEP, circular references and checkpointing failures

Hi Kostas, yes, I'm using 1.3.2. I'll try the current master and I'll get back to you. 2017-11-03 15:21 GMT+01:00 Kostas Kloudas : > Hi Federico, > > I assume that you are using Flink 1.3, right? > > In this case, in 1.4 we have fixed a bug that seems similar to your case: > https://issues.apach

Re: FlinkCEP, circular references and checkpointing failures

Hi Federico, I assume that you are using Flink 1.3, right? In this case, in 1.4 we have fixed a bug that seems similar to your case: https://issues.apache.org/jira/browse/FLINK-7756 Could you try the current master to see if it fixes your probl

Re: FlinkCEP, circular references and checkpointing failures

I'm sorry, I realized that the stacktrack was poorly formatted, here it is a better formatting: AsynchronousException{java.lang.Exception: Could not materialize checkpoint 2 for operator KeyedCEPPatternOperator -> alert-select -> Sink: notification-sink-1 (1/1).} at org.apache.flink.stream

FlinkCEP, circular references and checkpointing failures

Hello everyone, I'm a bit experimenting with FlinkCEP and I'm noticing weird failures when it comes to checkpoints and within clauses windows closing at the same time a (synchronous, both on Fs and RocksDB, stored in hdfs) checkpoint occurs. The following is the relevant code: val env : StreamEx

Re: Negative values using latency marker

Hi Tovi, if I see this correctly, the LatencyMarker gets its initial timstamp during creation at the source and the latency is reported as a metric at a sink by comparing the initial timestamp with the current time. If the clocks between the two machines involved diverge, e.g. the sinks clock fa

Re: Docker-Flink Project: TaskManagers can't talk to JobManager if they are on different nodes

Just an update: by changing the JOB_MANAGER_RPC_ADDRESS to the public IP of the JobManager and exposing port 6123 as {{PUBLIC_IP}}:6123:6123, I manged to get the TaskManagers from different nodes and even different subnets to talk to the JobManager. This is how I created the services: docker