Re: Kafka Streams throughput performance overtime

2018-07-24 Thread Guozhang Wang
Hello Siva, I'd suggest you upgrade to Kafka 2.0 once it is released (should be out soon, probably this week) as it includes a critical performance optimization for windowed aggregation operations. Note that even if your broker is in older versions, new versioned clients like Streams can still ta

Kafka Streams throughput performance overtime

2018-07-23 Thread Siva Ram
Hi, I have a stream application that performs rollup from 15mins to Hourly, then Hourly to Daily. The process needs to be continuously run 24 hours and each 15 mins approx 12 million records gets posted (a JSON record per message) into the input topic. There are 3 separate processors correspondi