Let's say a consumer wants to fetch 1000 bytes and there are two messages in the broker each of 600 bytes. The broker will give 1000 bytes to the consumer. The consumer will only be able to use the first 600 bytes since the remaining 400 bytes doesn't form a full message.
0.7 jmx are documented in https://cwiki.apache.org/confluence/display/KAFKA/Operations thanks, Jun On Tue, Apr 29, 2014 at 9:51 PM, Arnaud Lawson <arnaud.law...@velos.io>wrote: > Thank you Jun, > > What could a partial message be? A consumer fetching for messages on an > empty queue and receiving a message representing that fact? > > As an example below i just ran a test using the producer & consumer shells > and JMXTerm. NumProduceRequests went from 13, 14 to 15 as i added values > one by one to the queue for a test topic but then NumFetchRequests varies > greatly. I am not sure why. I know there is only one consumer (the shell > consumer) for this test kafka 0.7 instance and replication is not available > for this version. *Essentially** we would like to be able to measure the > rate of consumption for some particular topic and make sure that this rate > is about the same as the rate of production for that topic. Which JMX > metrics would be appropriate for this for Kafka 0.7?* Any suggestions would > be helpful. Thanks. > > > #mbean = kafka:type=kafka.SocketServerStats: > NumProduceRequests = 13; (Attribute exposed for management) > > $>get -i -b kafka:type=kafka.SocketServerStats NumFetchRequests > #mbean = kafka:type=kafka.SocketServerStats: > NumFetchRequests = 4763; (Attribute exposed for management) > > $>get -i -b kafka:type=kafka.SocketServerStats NumProduceRequests > #mbean = kafka:type=kafka.SocketServerStats: > NumProduceRequests = 14; (Attribute exposed for management) > > $>get -i -b kafka:type=kafka.SocketServerStats NumFetchRequests > #mbean = kafka:type=kafka.SocketServerStats: > NumFetchRequests = 4776; (Attribute exposed for management) > > $>get -i -b kafka:type=kafka.SocketServerStats NumProduceRequests > #mbean = kafka:type=kafka.SocketServerStats: > NumProduceRequests = 15; (Attribute exposed for management) > > $>get -i -b kafka:type=kafka.SocketServerStats NumFetchRequests > #mbean = kafka:type=kafka.SocketServerStats: > NumFetchRequests = 4904; (Attribute exposed for management) > > > > > On Wed, Apr 30, 2014 at 12:18 AM, Jun Rao <jun...@gmail.com> wrote: > > > Is the data consumed more than once from different consumers? If that's > not > > the case, byteout could be a bit larger than bytein since we could > return a > > partial message to the consumer. The partial message will be ignored in > the > > consumer though. > > > > Thanks, > > > > Jun > > > > > > On Tue, Apr 29, 2014 at 2:13 PM, Arnaud Lawson <arnaud.law...@velos.io > > >wrote: > > > > > We are using Kafka 0.7.1 and i believe replication isn't available for > > this > > > version (correct me if i am wrong)... > > > > > > Basically we want to be able to measure the rate of consumption for > some > > > particular topic and make sure that this rate is about the same as the > > rate > > > of production for that topic. Which JMX metrics could be appropriate > for > > > this - for Kafka 0.7? Thanks. > > > > > > > > > On Tue, Apr 29, 2014 at 4:28 PM, Bae, Jae Hyeon <metac...@gmail.com> > > > wrote: > > > > > > > Are you using 0.8? BytesOut will include the traffic for replication. > > If > > > > you have no consumer and replication factor is 2, BytesOut should be > > > > exactly double of BytesIn. > > > > > > > > > > > > On Tue, Apr 29, 2014 at 1:26 PM, Arnaud Lawson < > arnaud.law...@velos.io > > > > >wrote: > > > > > > > > > Hello, > > > > > > > > > > After graphing the cumulative values of Bytesin and Bytesout for > one > > of > > > > our > > > > > Kafka topics, we've noticed that the Bytesout values are sometimes > > > > greater > > > > > than the Bytesin values. We are not really sure how this would be > > > > possible > > > > > if the max that a consumer can consume would be what has been put > on > > > the > > > > > queue for that topic. Any explanations? Thanks. > > > > > > > > > > -- > > > > > > > > > > Arnaud Lawson > > > > > , > > > > > > > > > > Systems Operations Engineer > > > > > *VELOS* > > > > > Accelerating Machine Learning. > > > > > > > > > > 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001 > > > > > C: (240) 393 - 6703 > > > > > F: 646.349.4063 > > > > > E: arnaud.law...@velos.io > > > > > W: www.velos.io > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Arnaud Lawson > > > , > > > > > > Systems Operations Engineer > > > *VELOS* > > > Accelerating Machine Learning. > > > > > > 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001 > > > C: (240) 393 - 6703 > > > F: 646.349.4063 > > > E: arnaud.law...@velos.io > > > W: www.velos.io > > > > > > > > > -- > > Arnaud Lawson > , > > Systems Operations Engineer > *VELOS* > Accelerating Machine Learning. > > 440 9th AVE, 11TH FLOOR, NEW YORK, NY 10001 > C: (240) 393 - 6703 > F: 646.349.4063 > E: arnaud.law...@velos.io > W: www.velos.io >
