Hi David,
It did look like a bug.
Could you file a bug in JIRA?
And if you have time, welcome to investigate and submit PR for it.
(My guess is there are some internal topics are included in
`TotalFetchRequestsPerSec`, but not included in another one, but not sure)
Thank you.
Luke
On Sat, Feb 25, 2023 at 2:31 AM David Ballano Fernandez <
dfernan...@demonware.net> wrote:
> Hi Pere,
>
> Thanks for replying.
>
> so what i can see is:
>
> *kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec*
> and
> *kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce}*
> show the same/close number of producer requests and that makes sense.
>
> but that is not the same with fetch requests, I pull those metrics from
> Kafka jmx "oneMinuterate" without any other calculations. and this is what
> i see:
>
> kafka.server:type=BrokerTopicMetrics,name=TotalFetchRequestsPerSec
> *3.1Million
> req/s*
> kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Fetch}
> *97k req/s*
> I always trusted "
> *kafka.network:type=RequestMetrics,name=RequestsPerSec,request" *and I
> thought* "TotalFetchRequestsPerSec" *would be similar with the ability
> like you say to see per topic. but the difference is huge and I don't know
> what to trust.
>
> Thanks.
>
>
> On Mon, Feb 20, 2023 at 2:56 AM Pere Urbón Bayes <pere.ur...@gmail.com>
> wrote:
>
>> Hi David, these two metrics have two different objectives. > kafka.
>> server: type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec < ****
>> Produce request rate per topic. Omitting 'topic=(. . . )' will yield the
>> all-topic rate. *****while >kafka. network:
>> type=RequestMetrics,name=RequestsPerSec,request={Produce|FetchConsumer|FetchFollower}
>>
>> ZjQcmQRYFpfptBannerStart
>> This Message Is From an Untrusted Sender
>> You have not previously corresponded with this sender.
>>
>> ZjQcmQRYFpfptBannerEnd
>> Hi David,
>> these two metrics have two different objectives.
>>
>> > kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec <
>>
>> ****
>> Produce request rate per topic. Omitting 'topic=(...)' will yield the
>> all-topic rate.
>> *****
>>
>> while
>>
>> >kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce|FetchConsumer|FetchFollower}
>> <
>>
>> it is going to report as well requests per sec, however, you can filter
>> and select by Produce/Fetch from consumers or Fetch from Followers.
>> Note this is a metric collected at the network type, while the previous
>> is at the server level.
>>
>> The first one will provide you with info per topic, which is a very
>> valuable asset in order to know more about what kind of usage clients are
>> doing from a topic perspective. Keep in mind the second one, will only give
>> you the global request per second.
>>
>> So to your questions:
>>
>> > Shouldn't both metrics indicate the same?
>>
>> They indicate similar data at different levels, so they should not be far
>> away, however, some distance (but minimal) between the two could be
>> expected as they are collected at two different points.
>>
>> > seems to be the accurate one matching the number of messages sent with
>> no batching.
>>
>> In a good monitoring solution, you should have both metrics as one will
>> give you the overall load and the other the topic partition per broker view
>> (aka what are your users doing).
>>
>> what I don't understand is the batching part of your question.
>>
>> If you like to see messages In I would suggest you to use
>>
>> > kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec <
>> Aggregate incoming message rate.
>>
>> Sincerely
>>
>> -- Pere
>>
>>
>> On Sat, Feb 18, 2023 at 12:17 AM David Ballano Fernandez <
>> dfernan...@demonware.net> wrote:
>>
>>> Hi guys,
>>>
>>> I am having some confusion around 2 Kafka metrics:
>>>
>>> *Request rate.*
>>>
>>> kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce|FetchConsumer|FetchFollower}
>>>
>>> *Produce request rate.*
>>> kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec
>>>
>>>
>>> https://docs.confluent.io/platform/current/kafka/monitoring.html#server-metrics
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.confluent.io_platform_current_kafka_monitoring.html-23server-2Dmetrics&d=DwMFaQ&c=qE8EibqjfXM-zBfebVhd4gtjNZbrDcrKYXvb1gt38s4&r=p-f3AJg4e4Uk20g_16kSyBtabT4JOB-1GIb23_CxD58&m=S_eduaJSLvK0WMz0OiEppjB-dWgGa16RxmcEYZGwTvMMPf6CruRKu8aCf1_lk1L4&s=wHj2BcMsO0BRKhP_R042bVe3Vw9nhHFFWpB00gv2uVw&e=>
>>>
>>> They seem to be tracking the same, but when I graph them and say I pick
>>> Producer. The numbers are different.
>>>
>>> and looks like
>>>
>>> *Produce request rate.*
>>> kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec
>>>
>>> seems to be the accurate one matching the number of messages sent with no
>>> batching.
>>>
>>> Shouldn't both metrics indicate the same?
>>>
>>> Thanks!
>>>
>>
>>
>> --
>> Pere Urbon-Bayes
>> Software Architect
>> https://twitter.com/purbon
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_purbon&d=DwMFaQ&c=qE8EibqjfXM-zBfebVhd4gtjNZbrDcrKYXvb1gt38s4&r=p-f3AJg4e4Uk20g_16kSyBtabT4JOB-1GIb23_CxD58&m=S_eduaJSLvK0WMz0OiEppjB-dWgGa16RxmcEYZGwTvMMPf6CruRKu8aCf1_lk1L4&s=IeLld3jaDO53mdNUhmAdKzsjuxNI5z466ilSuvUOaFA&e=>
>> https://www.linkedin.com/in/purbon/
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_in_purbon_&d=DwMFaQ&c=qE8EibqjfXM-zBfebVhd4gtjNZbrDcrKYXvb1gt38s4&r=p-f3AJg4e4Uk20g_16kSyBtabT4JOB-1GIb23_CxD58&m=S_eduaJSLvK0WMz0OiEppjB-dWgGa16RxmcEYZGwTvMMPf6CruRKu8aCf1_lk1L4&s=KepVYe2cAVyV3SU0NUn2g5rLyuY6QWmunVA5xXUWDVE&e=>
>>
>
