Re: Flink - Iteration and Backpressure

2017-06-13 Thread MAHESH KUMAR
Hi Robert/Team, Is there any recommended solution or any other insight on how I must be doing it? Thanks and Regards, Mahesh On Thu, Jun 1, 2017 at 10:32 AM, MAHESH KUMAR wrote: > Hi Robert, > > The Message Auditor System must monitor all the 4 kafka queue and gather > information about messag

Re: Flink - Iteration and Backpressure

2017-06-01 Thread MAHESH KUMAR
Hi Robert, The Message Auditor System must monitor all the 4 kafka queue and gather information about messages that made through all of them or say specifically which queue a particular message did not make it through. We want the window time to be equivalent to our SLA time so that any message th

Re: Flink - Iteration and Backpressure

2017-06-01 Thread Robert Metzger
Hi Mahesh, why do you need to iterate over the elements? I wonder if you can't just stream the data from kafka1-kafka4 through your topology? On Fri, May 26, 2017 at 7:21 PM, MAHESH KUMAR wrote: > Hi Team, > > I am trying to build an audit like system where I read messages from "n" > Kafka q

Flink - Iteration and Backpressure

2017-05-26 Thread MAHESH KUMAR
Hi Team, I am trying to build an audit like system where I read messages from "n" Kafka queues, key by a unique key and then reduce them to a single message, if it has passed through all the "n" Kafka queues in a window time of "m" hours/days, the message has succeeded else it has expired. I can