Hi, I have a stream application that performs rollup from 15mins to Hourly, then Hourly to Daily. The process needs to be continuously run 24 hours and each 15 mins approx 12 million records gets posted (a JSON record per message) into the input topic. There are 3 separate processors corresponding to the above, where Hourly and Daily maintains the state. So, in hourly 10million needs to be retained ever hour in the state and in daily overall 10 million. 42GB of memory is allocated for the whole application and throughput is fine for until first 10hrs, after that it degrades significantly. Any suggestions on this to identify the delay and to increase the throughput would be of great help? We are on Kafka 1.0.0
Thanks Siva
