If you open up jmx (e.g. jconsole) in a consumer instance, you will see the jmx of name *-ConsumerLag.
Thanks, Jun On Thu, May 29, 2014 at 8:53 AM, Рябков Алексей Николаевич < a.ryab...@ntc-vulkan.ru> wrote: > Do you mean getOffsetLag ? ( > https://cwiki.apache.org/confluence/display/KAFKA/Operations) > > But its all about offset not message... > > So for example to create "real" load balance between broker I can: > > 1. Calculate average unread message in topic per broker: > - Sall=sum(kafka.BrokerAllTopicStat.[topic]. getMessagesIn) -total > send messages in topic for all broker > - Rall=sum (kafka.ConsumerTopicStat.[topic]. getMessagesPerTopic - > -total read messages in topic for all consumers > - (Sall-Rall)/broker number - average unread messages in topic per > broker > 2. Get current number of unread messages in topic for broker (little hack > here) > for example if we locate N consumers per broker and each consumer > must read only from 1 partition than we can : > - Sb=kafka.BrokerAllTopicStat.[topic]. getMessagesIn) -total send > messages in topic for broker B > - Rb=sum for all N consumers in broker b > (kafka.ConsumerTopicStat.[topic]. getMessagesPerTopic) - total unread > messages in topic for N consumers on broker b > - (Sb-Rb) -unread messages in topic for broker > 3. Use partitioner to load balance between brokers ... > - compare average unread messages in topic per broker with current > unread messages in topic for broker we can create more clever load > balancer... > > > But what can you tell about performance?... How fast I can get monitoring > stat? Could you give me some advice for optimization? > > > Thanks, Aleksey Ryabkov > > > -----Исходное сообщение----- > От: Jun Rao [mailto:jun...@gmail.com] > Отправлено: Thursday, May 29, 2014 8:34 AM > Кому: users@kafka.apache.org > Тема: Re: mBean to monitor message per partitions in topic > > There is a per-partition jmx (*-ConsumerLag) in the consumer that reports > unconsumed messages per partition. > > Thanks, > > Jun > > > On Wed, May 28, 2014 at 8:13 AM, Рябков Алексей Николаевич < > a.ryab...@ntc-vulkan.ru> wrote: > > > Hello! > > > > How can I get information about unfetched message per partition in topic? > > I wish to use such information to create my custom > > partitioner.class to balance messages between partitions > > > > With best regards, Aleksey Ryabkov > > > > >
