Kafka Streams throughput performance overtime

2018-07-23 Thread Siva Ram
Hi, I have a stream application that performs rollup from 15mins to Hourly, then Hourly to Daily. The process needs to be continuously run 24 hours and each 15 mins approx 12 million records gets posted (a JSON record per message) into the input topic. There are 3 separate processors correspondi

Kafka stream application load balancing

2018-07-29 Thread Siva Ram
Hi, Kafka version 1.0.0 (can't upgrade to another version yet due to legacy dependency) The stream application uses low level processor API and maintains state. A topic is setup with 30 partitions and I had split to 2 stream application instances consuming the same topic, each with 15 threads.

Re: Kafka stream application load balancing

2018-08-05 Thread Siva Ram
on state store restoration metrics ( > https://kafka.apache.org/documentation/#kafka_streams_store_monitoring) > such as "restore-latency-avg" and "restore-rate". 3) Look into your log4j > and check for "partition revocation took" and "partition assignment took"

Error persisting into KeyValueStore

2018-08-06 Thread Siva Ram
Hi, We switched to a virtual machine (from physical node) and we are observing the following exception occurs and all our the stream application instances fails. Can you please advise what causes this exception? 2018-08-07 01:31:14 ERROR MetaStream380 - Unknown Exception occurred org.apache.kaf