Thank you Apostolis Glenis. On Thu, Dec 14, 2017 at 7:30 PM, Apostolis Glenis <apostgle...@gmail.com> wrote:
> I have also created a monitoring application for Kafka that uses > prometheus. > You can look at the source code here: > > https://github.com/aglenis/kafka_monitoring_pandas > > 2017-12-13 9:53 GMT+02:00 Irtiza Ali <i...@an10.io>: > > > Ok thank you Michal > > > > On Tue, Dec 12, 2017 at 9:30 PM, Michal Michalski < > > michal.michal...@zalando.ie> wrote: > > > > > Hi Irtiza, > > > > > > Unfortunately I don't know what could be the issue in your case - in my > > > case all these metrics simply appear under appropriate keys > > > (e.g. "kafka.controller") in the JSON response of the /list request - > the > > > same as the one you're making (I checked it right now to be sure). I > also > > > checked our Kafka broker configuration and we don't seem to use any > > options > > > related to metrics reporting (I don't even know if there are any...). > > > Reading the Kafka documentation I'm under impression that all the > metrics > > > should be available to you out-of-the-box, so I'm not sure why you > can't > > > see them, sorry :-( > > > > > > M. > > > > > > > > > On 11 December 2017 at 14:11, Irtiza Ali <i...@an10.io> wrote: > > > > > > > Hi Michal, > > > > > > > > I have followed the steps below for the configuring Jolokia with > kafka > > > > > > > > - > > > > > > > > Install the jolokia JVM-Agent jar from their site. > > > > - > > > > > > > > Copy the jolokia jar file in the /opt/jolokia-agent/, create > > > > jolokia-agent > > > > directory as it doesn’t exist already. > > > > - > > > > > > > > Move to the directory where your Kafka exists run the following > > > command > > > > > > > > $ export KAFKA_OPTS="$KAFKA_OPTS > > > > -javaagent:/opt/jolokia-agent/jolokia-jvm-1.3.7-agent.jar" > > > > > > > > - > > > > > > > > To expose the kafka metrics open the this file > > > > /kafka/bin/kafka-server-start.sh file and find this (exec > > > > $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@") line > and > > > > paste > > > > the line given below above it. > > > > > > > > > > > > export JMX_PORT=${JMX_PORT:-9999} > > > > > > > > - > > > > > > > > Restart the zookeeper and broker > > > > > > > > > > > > I think the issue is with the way I am exposing the metrics. I > followed > > > the > > > > following link to enable JMX on kafka broker ( > > > > https://stackoverflow.com/questions/36708384/enable-jmx- > > on-kafka-brokers > > > ). > > > > The issue that I am still facing is that when I list all the metrics > > > using > > > > ( > > > > http://localhost:9999/jolokia/list) address. i dont get metrics > > related > > > > to: > > > > <host>:8074/jolokia/read/kafka.server:* > > > > <host>:8074/jolokia/read/kafka.controller:* > > > > <host>:8074/jolokia/read/kafka.log:* > > > > <host>:8074/jolokia/read/kafka.network:* > > > > > > > > > > > > Can you check what I have done wrong? > > > > > > > > Thanks > > > > > > > > With Regards > > > > Irtiza > > > > > > > > On Fri, Dec 8, 2017 at 8:19 PM, Michal Michalski < > > > > michal.michal...@zalando.ie> wrote: > > > > > > > > > Hi, > > > > > > > > > > We have no modifications in that file - what we do is having a > > > "wrapper" > > > > > that's just a Docker "entrypoint" (just a bash script) which > contents > > > is: > > > > > > > > > > export KAFKA_OPTS="$KAFKA_OPTS > > > > > -javaagent:/jolokia-jvm-agent.jar=port=8074,host=0.0.0.0" > > > > > exec ${KAFKA_DIR}/bin/kafka-server-start.sh > > > > > ${KAFKA_DIR}/config/server.properties > > > > > > > > > > Regarding your second question - we have an in-house monitoring > > > solution > > > > > that allows querying that endpoint and extracting metrics from the > > JSON > > > > > returned. > > > > > > > > > > If you're asking about what you should monitor, I think these links > > > will > > > > > answer your question: > > > > > https://docs.confluent.io/3.0.0/kafka/monitoring.html > > > > > https://www.datadoghq.com/blog/monitoring-kafka- > performance-metrics/ > > > > > > > > > > > > > > > On 8 December 2017 at 14:56, Irtiza Ali <i...@an10.io> wrote: > > > > > > > > > > > Thanks Michal, can you kindly send me you kafka-run-class.sh and > > > > > > kafka-server-start.sh file, I have look what have you done. > > Because I > > > > > have > > > > > > done same thing that you explained above but when i do this < > > > > > > http://localhost:9999/jolokia/list> i get only metrics for the > > > > zookeeper > > > > > > but not the above metrics. > > > > > > > > > > > > How are you using Jolokia for monitoring kafka cluster?? > > > > > > > > > > > > Thanks in advance > > > > > > > > > > > > On Fri, Dec 8, 2017 at 3:10 PM, Michal Michalski < > > > > > > michal.michal...@zalando.ie> wrote: > > > > > > > > > > > > > Hi Irtiza, > > > > > > > > > > > > > > I don't have any tutorial, but I can tell you what we do :-) > > > > > > > > > > > > > > First of all we have Jolokia agent jar included in our Kafka > > Docker > > > > > > image. > > > > > > > Then we append this to KAFKA_OPTS > > > > > > > > > > > > > > -javaagent:/jolokia-jvm-agent.jar=port=8074,host=0.0.0.0 > > > > > > > > > > > > > > > > > > > > > Relevant port is then "exposed" in Docker image and "allowed" > in > > > AWS > > > > > > > Security Group. > > > > > > > Then our monitoring tool is querying the following endpoins > > (just a > > > > > HTTP > > > > > > > GET query) these endpoints to get a list of all the metrics we > > > need: > > > > > > > > > > > > > > <host>:8074/jolokia/read/kafka.server:* > > > > > > > <host>:8074/jolokia/read/kafka.controller:* > > > > > > > <host>:8074/jolokia/read/kafka.log:* > > > > > > > <host>:8074/jolokia/read/kafka.network:* > > > > > > > <host>:8074/jolokia/read/java.lang:type=Memory > > > > > > > > > > > > > > > > > > > > > And that's it, we get a nice JSON that we can use the way we > want > > > :-) > > > > > > > I hope I didn't miss anything, but this should be it. > > > > > > > > > > > > > > M. > > > > > > > > > > > > > > > > > > > > > On 8 December 2017 at 09:28, Irtiza Ali <i...@an10.io> wrote: > > > > > > > > > > > > > > > Hello Michal, > > > > > > > > > > > > > > > > Can you send me link to tutorial or provide some resources > for > > > the > > > > > > > Jolokia > > > > > > > > configuration with kafka> > > > > > > > > > > > > > > > > > > > > > > > > Thank you > > > > > > > > Irtiza > > > > > > > > > > > > > > > > On Wed, Dec 6, 2017 at 8:00 PM, Michal Michalski < > > > > > > > > michal.michal...@zalando.ie> wrote: > > > > > > > > > > > > > > > > > Hi Irtiza, > > > > > > > > > > > > > > > > > > We're using Jolokia and we had no problems with it. > > > > > > > > > It would be useful to know what exactly you did (how you > > > "plugged > > > > > in" > > > > > > > > > Jolokia, how you configured it, what endpoint are you > > querying > > > > > etc.) > > > > > > to > > > > > > > > > help you. > > > > > > > > > > > > > > > > > > On 6 December 2017 at 10:36, Irtiza Ali <i...@an10.io> > > wrote: > > > > > > > > > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > > > > > > > > > I am working python based Kafka monitoring application. I > > am > > > > > unable > > > > > > > to > > > > > > > > > > figure out how to retrieve the metrics using Jolokia. I > > have > > > > > enable > > > > > > > the > > > > > > > > > > port for metrics retrieval to 9999. > > > > > > > > > > > > > > > > > > > > I Have two questions > > > > > > > > > > > > > > > > > > > > 1) Is there something that I am not doing correctly. > > > > > > > > > > 2) Is there some other way to do it. > > > > > > > > > > > > > > > > > > > > With Regards > > > > > > > > > > Irtiza Alli > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
