HA: mBean to monitor message per partitions in topic

[Рябков Алексей Николаевич]

Do you mean getOffsetLag ? 
(https://cwiki.apache.org/confluence/display/KAFKA/Operations)

But its all about offset not message...

So for example to create "real" load balance between broker I can:

1. Calculate average unread message in topic per broker:
       - Sall=sum(kafka.BrokerAllTopicStat.[topic]. getMessagesIn) -total send 
messages in topic for all broker
       - Rall=sum (kafka.ConsumerTopicStat.[topic]. getMessagesPerTopic -  
-total read messages in topic for all consumers
       - (Sall-Rall)/broker number - average unread messages in topic per 
broker 
2. Get current number of unread messages in topic for broker (little hack here)
      for example if we  locate N consumers per broker and each consumer must 
read only from 1 partition than we can :
       - Sb=kafka.BrokerAllTopicStat.[topic]. getMessagesIn) -total send 
messages in topic for broker B
       - Rb=sum for all N consumers in broker b 
(kafka.ConsumerTopicStat.[topic]. getMessagesPerTopic) - total unread messages 
in topic for N consumers on broker b
        - (Sb-Rb) -unread messages in topic for broker
3. Use partitioner to load balance between brokers ...
        - compare average unread messages in topic per broker with current 
unread messages in topic for broker we can create more clever load balancer...
   

But what can you tell about performance?... How fast I can get monitoring stat? 
 Could you give me some advice for optimization?
 

Thanks, Aleksey Ryabkov


-----Исходное сообщение-----
От: Jun Rao [mailto:jun...@gmail.com] 
Отправлено: Thursday, May 29, 2014 8:34 AM
Кому: users@kafka.apache.org
Тема: Re: mBean to monitor message per partitions in topic

There is a per-partition jmx (*-ConsumerLag) in the consumer that reports 
unconsumed messages per partition.

Thanks,

Jun


On Wed, May 28, 2014 at 8:13 AM, Рябков Алексей Николаевич < 
a.ryab...@ntc-vulkan.ru> wrote:

> Hello!
>
> How can I get information about unfetched message per partition in topic?
>        I wish to use such information to create my custom 
> partitioner.class to balance messages between partitions
>
> With best regards, Aleksey Ryabkov
>
>