Hi Irtiza,
Unfortunately I don't know what could be the issue in your case - in my
case all these metrics simply appear under appropriate keys
(e.g. "kafka.controller") in the JSON response of the /list request - the
same as the one you're making (I checked it right now to be sure). I also
checked our Kafka broker configuration and we don't seem to use any options
related to metrics reporting (I don't even know if there are any...).
Reading the Kafka documentation I'm under impression that all the metrics
should be available to you out-of-the-box, so I'm not sure why you can't
see them, sorry :-(
M.
On 11 December 2017 at 14:11, Irtiza Ali <i...@an10.io> wrote:
> Hi Michal,
>
> I have followed the steps below for the configuring Jolokia with kafka
>
> -
>
> Install the jolokia JVM-Agent jar from their site.
> -
>
> Copy the jolokia jar file in the /opt/jolokia-agent/, create
> jolokia-agent
> directory as it doesn’t exist already.
> -
>
> Move to the directory where your Kafka exists run the following command
>
> $ export KAFKA_OPTS="$KAFKA_OPTS
> -javaagent:/opt/jolokia-agent/jolokia-jvm-1.3.7-agent.jar"
>
> -
>
> To expose the kafka metrics open the this file
> /kafka/bin/kafka-server-start.sh file and find this (exec
> $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@") line and
> paste
> the line given below above it.
>
>
> export JMX_PORT=${JMX_PORT:-9999}
>
> -
>
> Restart the zookeeper and broker
>
>
> I think the issue is with the way I am exposing the metrics. I followed the
> following link to enable JMX on kafka broker (
> https://stackoverflow.com/questions/36708384/enable-jmx-on-kafka-brokers).
> The issue that I am still facing is that when I list all the metrics using
> (
> http://localhost:9999/jolokia/list) address. i dont get metrics related
> to:
> <host>:8074/jolokia/read/kafka.server:*
> <host>:8074/jolokia/read/kafka.controller:*
> <host>:8074/jolokia/read/kafka.log:*
> <host>:8074/jolokia/read/kafka.network:*
>
>
> Can you check what I have done wrong?
>
> Thanks
>
> With Regards
> Irtiza
>
> On Fri, Dec 8, 2017 at 8:19 PM, Michal Michalski <
> michal.michal...@zalando.ie> wrote:
>
> > Hi,
> >
> > We have no modifications in that file - what we do is having a "wrapper"
> > that's just a Docker "entrypoint" (just a bash script) which contents is:
> >
> > export KAFKA_OPTS="$KAFKA_OPTS
> > -javaagent:/jolokia-jvm-agent.jar=port=8074,host=0.0.0.0"
> > exec ${KAFKA_DIR}/bin/kafka-server-start.sh
> > ${KAFKA_DIR}/config/server.properties
> >
> > Regarding your second question - we have an in-house monitoring solution
> > that allows querying that endpoint and extracting metrics from the JSON
> > returned.
> >
> > If you're asking about what you should monitor, I think these links will
> > answer your question:
> > https://docs.confluent.io/3.0.0/kafka/monitoring.html
> > https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics/
> >
> >
> > On 8 December 2017 at 14:56, Irtiza Ali <i...@an10.io> wrote:
> >
> > > Thanks Michal, can you kindly send me you kafka-run-class.sh and
> > > kafka-server-start.sh file, I have look what have you done. Because I
> > have
> > > done same thing that you explained above but when i do this <
> > > http://localhost:9999/jolokia/list> i get only metrics for the
> zookeeper
> > > but not the above metrics.
> > >
> > > How are you using Jolokia for monitoring kafka cluster??
> > >
> > > Thanks in advance
> > >
> > > On Fri, Dec 8, 2017 at 3:10 PM, Michal Michalski <
> > > michal.michal...@zalando.ie> wrote:
> > >
> > > > Hi Irtiza,
> > > >
> > > > I don't have any tutorial, but I can tell you what we do :-)
> > > >
> > > > First of all we have Jolokia agent jar included in our Kafka Docker
> > > image.
> > > > Then we append this to KAFKA_OPTS
> > > >
> > > > -javaagent:/jolokia-jvm-agent.jar=port=8074,host=0.0.0.0
> > > >
> > > >
> > > > Relevant port is then "exposed" in Docker image and "allowed" in AWS
> > > > Security Group.
> > > > Then our monitoring tool is querying the following endpoins (just a
> > HTTP
> > > > GET query) these endpoints to get a list of all the metrics we need:
> > > >
> > > > <host>:8074/jolokia/read/kafka.server:*
> > > > <host>:8074/jolokia/read/kafka.controller:*
> > > > <host>:8074/jolokia/read/kafka.log:*
> > > > <host>:8074/jolokia/read/kafka.network:*
> > > > <host>:8074/jolokia/read/java.lang:type=Memory
> > > >
> > > >
> > > > And that's it, we get a nice JSON that we can use the way we want :-)
> > > > I hope I didn't miss anything, but this should be it.
> > > >
> > > > M.
> > > >
> > > >
> > > > On 8 December 2017 at 09:28, Irtiza Ali <i...@an10.io> wrote:
> > > >
> > > > > Hello Michal,
> > > > >
> > > > > Can you send me link to tutorial or provide some resources for the
> > > > Jolokia
> > > > > configuration with kafka>
> > > > >
> > > > >
> > > > > Thank you
> > > > > Irtiza
> > > > >
> > > > > On Wed, Dec 6, 2017 at 8:00 PM, Michal Michalski <
> > > > > michal.michal...@zalando.ie> wrote:
> > > > >
> > > > > > Hi Irtiza,
> > > > > >
> > > > > > We're using Jolokia and we had no problems with it.
> > > > > > It would be useful to know what exactly you did (how you "plugged
> > in"
> > > > > > Jolokia, how you configured it, what endpoint are you querying
> > etc.)
> > > to
> > > > > > help you.
> > > > > >
> > > > > > On 6 December 2017 at 10:36, Irtiza Ali <i...@an10.io> wrote:
> > > > > >
> > > > > > > Hello everyone,
> > > > > > >
> > > > > > > I am working python based Kafka monitoring application. I am
> > unable
> > > > to
> > > > > > > figure out how to retrieve the metrics using Jolokia. I have
> > enable
> > > > the
> > > > > > > port for metrics retrieval to 9999.
> > > > > > >
> > > > > > > I Have two questions
> > > > > > >
> > > > > > > 1) Is there something that I am not doing correctly.
> > > > > > > 2) Is there some other way to do it.
> > > > > > >
> > > > > > > With Regards
> > > > > > > Irtiza Alli
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
