Streaming job gets slower and slower

2017-08-27 Thread Aparup Banerjee (apbanerj)
We have a flink job with 7 subtasks. See graph below. This is on flink 1.2. [cid:image001.png@01D31EC7.FB4B5620] Here each source task consumes from a kafka topic. Data rate is low around 70-80 messages per sec. What we are noticing is after running for 2 hours or so the source tasks starts sho

FLINK-2821 - Flink on Kubernetes

2016-11-17 Thread Aparup Banerjee (apbanerj)
Hi Max, I am running into an issue on running flink on Kubernetes – basically during submission of a job. It looks like akka related. I then ran into this Jira FLINK-2821. I am not using flink directly , but I embed flink streaming and runtime jars as part a Java app. Would you have a Git PR whic

Re: Apache siddhi into Flink

2016-08-29 Thread Aparup Banerjee (apbanerj)
lved in the processing? Regards, Chesnay On 28.08.2016 23:21, Aparup Banerjee (apbanerj) wrote: Sorry for the semantic difference. On Aug 28, 2016, at 12:05 PM, Trevor Grant <<mailto:trevor.d.gr...@gmail.com>trevor.d.gr...@gmail.com<mailto:trevor.d.gr...@gmail.com>> wrot

Re: Apache siddhi into Flink

2016-08-28 Thread Aparup Banerjee (apbanerj)
uses of things." -Virgil On Sat, Aug 27, 2016 at 5:36 PM, Chen Qin mailto:qinnc...@gmail.com>> wrote: ?+1? On Aug 26, 2016, at 11:23 PM, Aparup Banerjee (apbanerj) mailto:apban...@cisco.com>> wrote: Hi- Has anyone looked into embedding apache siddhi into Flink. Thanks, Aparup

Apache siddhi into Flink

2016-08-26 Thread Aparup Banerjee (apbanerj)
Hi- Has anyone looked into embedding apache siddhi into Flink. Thanks, Aparup

Re: Supporting Multi-tenancy in a Flink

2016-07-26 Thread Aparup Banerjee (apbanerj)
Thanks. Hi Gyula anything , you can share on this? Aparup On 7/26/16, 4:38 AM, "Ufuk Celebi" wrote: >On Mon, Jul 25, 2016 at 5:38 AM, Aparup Banerjee (apbanerj) > wrote: >> We are building a Stream processing system using Apache beam on top of Flink >>

Supporting Multi-tenancy in a Flink

2016-07-24 Thread Aparup Banerjee (apbanerj)
We are building a Stream processing system using Apache beam on top of Flink using the Flink Runner. Our pipelines take Kafka streams as sources , and can write to multiple sinks. The system needs to be tenant aware. Tenants can share same Kafka topic. Tenants can write their own pipelines. We a