Thank you Jun, What could a partial message be? A consumer fetching for messages on an empty queue and receiving a message representing that fact?
As an example below i just ran a test using the producer & consumer shells and JMXTerm. NumProduceRequests went from 13, 14 to 15 as i added values one by one to the queue for a test topic but then NumFetchRequests varies greatly. I am not sure why. I know there is only one consumer (the shell consumer) for this test kafka 0.7 instance and replication is not available for this version. *Essentially** we would like to be able to measure the rate of consumption for some particular topic and make sure that this rate is about the same as the rate of production for that topic. Which JMX metrics would be appropriate for this for Kafka 0.7?* Any suggestions would be helpful. Thanks. #mbean = kafka:type=kafka.SocketServerStats: NumProduceRequests = 13; (Attribute exposed for management) $>get -i -b kafka:type=kafka.SocketServerStats NumFetchRequests #mbean = kafka:type=kafka.SocketServerStats: NumFetchRequests = 4763; (Attribute exposed for management) $>get -i -b kafka:type=kafka.SocketServerStats NumProduceRequests #mbean = kafka:type=kafka.SocketServerStats: NumProduceRequests = 14; (Attribute exposed for management) $>get -i -b kafka:type=kafka.SocketServerStats NumFetchRequests #mbean = kafka:type=kafka.SocketServerStats: NumFetchRequests = 4776; (Attribute exposed for management) $>get -i -b kafka:type=kafka.SocketServerStats NumProduceRequests #mbean = kafka:type=kafka.SocketServerStats: NumProduceRequests = 15; (Attribute exposed for management) $>get -i -b kafka:type=kafka.SocketServerStats NumFetchRequests #mbean = kafka:type=kafka.SocketServerStats: NumFetchRequests = 4904; (Attribute exposed for management) On Wed, Apr 30, 2014 at 12:18 AM, Jun Rao <jun...@gmail.com> wrote: > Is the data consumed more than once from different consumers? If that's not > the case, byteout could be a bit larger than bytein since we could return a > partial message to the consumer. The partial message will be ignored in the > consumer though. > > Thanks, > > Jun > > > On Tue, Apr 29, 2014 at 2:13 PM, Arnaud Lawson <arnaud.law...@velos.io > >wrote: > > > We are using Kafka 0.7.1 and i believe replication isn't available for > this > > version (correct me if i am wrong)... > > > > Basically we want to be able to measure the rate of consumption for some > > particular topic and make sure that this rate is about the same as the > rate > > of production for that topic. Which JMX metrics could be appropriate for > > this - for Kafka 0.7? Thanks. > > > > > > On Tue, Apr 29, 2014 at 4:28 PM, Bae, Jae Hyeon <metac...@gmail.com> > > wrote: > > > > > Are you using 0.8? BytesOut will include the traffic for replication. > If > > > you have no consumer and replication factor is 2, BytesOut should be > > > exactly double of BytesIn. > > > > > > > > > On Tue, Apr 29, 2014 at 1:26 PM, Arnaud Lawson <arnaud.law...@velos.io > > > >wrote: > > > > > > > Hello, > > > > > > > > After graphing the cumulative values of Bytesin and Bytesout for one > of > > > our > > > > Kafka topics, we've noticed that the Bytesout values are sometimes > > > greater > > > > than the Bytesin values. We are not really sure how this would be > > > possible > > > > if the max that a consumer can consume would be what has been put on > > the > > > > queue for that topic. Any explanations? Thanks. > > > > > > > > -- > > > > > > > > Arnaud Lawson > > > > , > > > > > > > > Systems Operations Engineer > > > > *VELOS* > > > > Accelerating Machine Learning. > > > > > > > > 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001 > > > > C: (240) 393 - 6703 > > > > F: 646.349.4063 > > > > E: arnaud.law...@velos.io > > > > W: www.velos.io > > > > > > > > > > > > > > > -- > > > > Arnaud Lawson > > , > > > > Systems Operations Engineer > > *VELOS* > > Accelerating Machine Learning. > > > > 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001 > > C: (240) 393 - 6703 > > F: 646.349.4063 > > E: arnaud.law...@velos.io > > W: www.velos.io > > > -- Arnaud Lawson , Systems Operations Engineer *VELOS* Accelerating Machine Learning. 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001 C: (240) 393 - 6703 F: 646.349.4063 E: arnaud.law...@velos.io W: www.velos.io
