Re: Need some help in identifying some important metrics to monitor for streams

Answer inline: > > I just wanted to understand say in single poll request if it fetches n > records does the above values indicate time computed for all n records or > just a single record. > In 0.10.2, the process latency is that of a single record, not the sum of n records. The commit latenc

Re: Need some help in identifying some important metrics to monitor for streams

Yes setting client id works. Now we are able to add metrics as part of our cron job. One additional question I have is http://kafka.apache.org/documentation.html#kafka_streams_monitoring I am monitoring the commit latency and process latency. Commit latency is usually say in 1000ms and process la

Re: Need some help in identifying some important metrics to monitor for streams

That is right, since client-id is used as the metrics name which should be distinguishable. https://kafka.apache.org/documentation/#streamsconfigs (I think we can improve on the explanation of the client.id config) A common client-id could contain the machine's host-port; of course, if you have m

Re: Need some help in identifying some important metrics to monitor for streams

Son if I am running my stream and across a cluster of different machine each machine should have a different client id. On 4 Mar 2017 12:36 a.m., "Guozhang Wang" wrote: > Sachin, > > The reason that you got metrics name as > > new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-StreamThread-1 >

Re: Need some help in identifying some important metrics to monitor for streams

Sachin, The reason that you got metrics name as new-part-advice-d1094e71-0f59-45e8-98f4-477f9444aa91-StreamThread-1 Is that you did not set the "CLIENT_ID_CONFIG" in your app, and KafkaStreams have to use a default combo of "appID: new-part-advice"-"processID: a UUID to guarantee uniqueness acr

Re: Need some help in identifying some important metrics to monitor for streams

Hi Sachin, Now that the confluent platform 3.2 is out, we also have some more documentation on this here: http://docs.confluent.io/3.2.0/streams/monitoring.html . We added a note on how to add other metrics. Yeah, your calculation on pol

Re: Need some help in identifying some important metrics to monitor for streams

Hi, I had checked the monitoring docs, but could not figure out which metrics are important ones. Also mainly I am looking at the average time spent between 2 successive poll requests. Can I say that average time between 2 poll requests is sum of commit + poll + process + punctuate (latency-avg).

Re: Need some help in identifying some important metrics to monitor for streams

Hi Sachin, The new streams metrics are now documented at https://kafka.apache.org/documentation/#kafka_streams_monitoring . Note that not all of them are turned on by default. We have several benchmarks that run nightly to moni