Hi All, I am running a kafka streaming application with a simple pipeline of: source topic -> group -> aggregate by key -> for each > save to a sink.
I source topic gets message at rate of 5000 - 10000 messages per second. During peak load we see the delay reaching to 3 million messages. So I need to figure out where delay might be happening. 1. Is there any mechanism in kafka streams to log time spent within each pipeline stage. 2. Also if I want to turn on custom logging to log some times how can I do the same. I have a log4j.properties and I am packaging it inside a jar which has the main class. I place that jar in libs folder of kafka installation. However I see no logs generated under logs folder. So where are we suppose to add the log4j.properties. Thanks Sachin
