In 0.8, we have a MaxLag jmx metrics to monitor the consumer lag. In 0.7, we only have a command line tool ConsumerOffsetChecker.
Thanks, Jun On Wed, Apr 30, 2014 at 7:17 AM, Arnaud Lawson <arnaud.law...@velos.io>wrote: > Jun, > > Thanks for the pointer to the docs. It would be great if more detail like > this discussion were included in the Wiki. The detail of each metric is not > always clear. > > For our case (trying to monitor and alert on if the consumer is falling far > behind the producer on a specific topic), what would you recommend? > > Should we compare BrokerAllTopicStat.[topic].getMessagesIn to > ConsumerTopicStat.[topic].getMessagesPerTopic? > > > On Wed, Apr 30, 2014 at 1:13 AM, Jun Rao <jun...@gmail.com> wrote: > > > Let's say a consumer wants to fetch 1000 bytes and there are two messages > > in the broker each of 600 bytes. The broker will give 1000 bytes to the > > consumer. The consumer will only be able to use the first 600 bytes since > > the remaining 400 bytes doesn't form a full message. > > > > 0.7 jmx are documented in > > https://cwiki.apache.org/confluence/display/KAFKA/Operations > > > > thanks, > > > > Jun > > > > > > On Tue, Apr 29, 2014 at 9:51 PM, Arnaud Lawson <arnaud.law...@velos.io > > >wrote: > > > > > Thank you Jun, > > > > > > What could a partial message be? A consumer fetching for messages on an > > > empty queue and receiving a message representing that fact? > > > > > > As an example below i just ran a test using the producer & consumer > > shells > > > and JMXTerm. NumProduceRequests went from 13, 14 to 15 as i added > values > > > one by one to the queue for a test topic but then NumFetchRequests > varies > > > greatly. I am not sure why. I know there is only one consumer (the > shell > > > consumer) for this test kafka 0.7 instance and replication is not > > available > > > for this version. *Essentially** we would like to be able to measure > the > > > rate of consumption for some particular topic and make sure that this > > rate > > > is about the same as the rate of production for that topic. Which JMX > > > metrics would be appropriate for this for Kafka 0.7?* Any suggestions > > would > > > be helpful. Thanks. > > > > > > > > > #mbean = kafka:type=kafka.SocketServerStats: > > > NumProduceRequests = 13; (Attribute exposed for management) > > > > > > $>get -i -b kafka:type=kafka.SocketServerStats NumFetchRequests > > > #mbean = kafka:type=kafka.SocketServerStats: > > > NumFetchRequests = 4763; (Attribute exposed for management) > > > > > > $>get -i -b kafka:type=kafka.SocketServerStats NumProduceRequests > > > #mbean = kafka:type=kafka.SocketServerStats: > > > NumProduceRequests = 14; (Attribute exposed for management) > > > > > > $>get -i -b kafka:type=kafka.SocketServerStats NumFetchRequests > > > #mbean = kafka:type=kafka.SocketServerStats: > > > NumFetchRequests = 4776; (Attribute exposed for management) > > > > > > $>get -i -b kafka:type=kafka.SocketServerStats NumProduceRequests > > > #mbean = kafka:type=kafka.SocketServerStats: > > > NumProduceRequests = 15; (Attribute exposed for management) > > > > > > $>get -i -b kafka:type=kafka.SocketServerStats NumFetchRequests > > > #mbean = kafka:type=kafka.SocketServerStats: > > > NumFetchRequests = 4904; (Attribute exposed for management) > > > > > > > > > > > > > > > On Wed, Apr 30, 2014 at 12:18 AM, Jun Rao <jun...@gmail.com> wrote: > > > > > > > Is the data consumed more than once from different consumers? If > that's > > > not > > > > the case, byteout could be a bit larger than bytein since we could > > > return a > > > > partial message to the consumer. The partial message will be ignored > in > > > the > > > > consumer though. > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > > > > > On Tue, Apr 29, 2014 at 2:13 PM, Arnaud Lawson < > arnaud.law...@velos.io > > > > >wrote: > > > > > > > > > We are using Kafka 0.7.1 and i believe replication isn't available > > for > > > > this > > > > > version (correct me if i am wrong)... > > > > > > > > > > Basically we want to be able to measure the rate of consumption for > > > some > > > > > particular topic and make sure that this rate is about the same as > > the > > > > rate > > > > > of production for that topic. Which JMX metrics could be > appropriate > > > for > > > > > this - for Kafka 0.7? Thanks. > > > > > > > > > > > > > > > On Tue, Apr 29, 2014 at 4:28 PM, Bae, Jae Hyeon < > metac...@gmail.com> > > > > > wrote: > > > > > > > > > > > Are you using 0.8? BytesOut will include the traffic for > > replication. > > > > If > > > > > > you have no consumer and replication factor is 2, BytesOut should > > be > > > > > > exactly double of BytesIn. > > > > > > > > > > > > > > > > > > On Tue, Apr 29, 2014 at 1:26 PM, Arnaud Lawson < > > > arnaud.law...@velos.io > > > > > > >wrote: > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > After graphing the cumulative values of Bytesin and Bytesout > for > > > one > > > > of > > > > > > our > > > > > > > Kafka topics, we've noticed that the Bytesout values are > > sometimes > > > > > > greater > > > > > > > than the Bytesin values. We are not really sure how this would > be > > > > > > possible > > > > > > > if the max that a consumer can consume would be what has been > put > > > on > > > > > the > > > > > > > queue for that topic. Any explanations? Thanks. > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Arnaud Lawson > > > > > > > , > > > > > > > > > > > > > > Systems Operations Engineer > > > > > > > *VELOS* > > > > > > > Accelerating Machine Learning. > > > > > > > > > > > > > > 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001 > > > > > > > C: (240) 393 - 6703 > > > > > > > F: 646.349.4063 > > > > > > > E: arnaud.law...@velos.io > > > > > > > W: www.velos.io > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Arnaud Lawson > > > > > , > > > > > > > > > > Systems Operations Engineer > > > > > *VELOS* > > > > > Accelerating Machine Learning. > > > > > > > > > > 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001 > > > > > C: (240) 393 - 6703 > > > > > F: 646.349.4063 > > > > > E: arnaud.law...@velos.io > > > > > W: www.velos.io > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Arnaud Lawson > > > , > > > > > > Systems Operations Engineer > > > *VELOS* > > > Accelerating Machine Learning. > > > > > > 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001 > > > C: (240) 393 - 6703 > > > F: 646.349.4063 > > > E: arnaud.law...@velos.io > > > W: www.velos.io > > > > > > > > > -- > > Arnaud Lawson > , > > Systems Operations Engineer > *VELOS* > Accelerating Machine Learning. > > 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001 > C: (240) 393 - 6703 > F: 646.349.4063 > E: arnaud.law...@velos.io > W: www.velos.io >
