Streaming job gets slower and slower

2017-08-27 Thread Aparup Banerjee (apbanerj)
We have a flink job with 7 subtasks. See graph below. This is on flink 1.2. [cid:image001.png@01D31EC7.FB4B5620] Here each source task consumes from a kafka topic. Data rate is low around 70-80 messages per sec. What we are noticing is after running for 2 hours or so the source tasks starts sho

Apache siddhi into Flink

2016-08-26 Thread Aparup Banerjee (apbanerj)
Hi- Has anyone looked into embedding apache siddhi into Flink. Thanks, Aparup

Supporting Multi-tenancy in a Flink

2016-07-24 Thread Aparup Banerjee (apbanerj)
We are building a Stream processing system using Apache beam on top of Flink using the Flink Runner. Our pipelines take Kafka streams as sources , and can write to multiple sinks. The system needs to be tenant aware. Tenants can share same Kafka topic. Tenants can write their own pipelines. We a