Hi Pere,
Thanks for replying.
so what i can see is:
*kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec* and
*kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce}*
show the same/close number of producer requests and that makes sense.
but that is not the same with fetch requests, I pull those metrics from
Kafka jmx "oneMinuterate" without any other calculations. and this is what
i see:
kafka.server:type=BrokerTopicMetrics,name=TotalFetchRequestsPerSec
*3.1Million
req/s*
kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Fetch}
*97k req/s*
I always trusted "
*kafka.network:type=RequestMetrics,name=RequestsPerSec,request" *and I
thought* "TotalFetchRequestsPerSec" *would be similar with the ability like
you say to see per topic. but the difference is huge and I don't know what
to trust.
Thanks.
On Mon, Feb 20, 2023 at 2:56 AM Pere Urbón Bayes <pere.ur...@gmail.com>
wrote:
> Hi David, these two metrics have two different objectives. > kafka.
> server: type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec < ****
> Produce request rate per topic. Omitting 'topic=(. . . )' will yield the
> all-topic rate. *****while >kafka. network:
> type=RequestMetrics,name=RequestsPerSec,request={Produce|FetchConsumer|FetchFollower}
>
> ZjQcmQRYFpfptBannerStart
> This Message Is From an Untrusted Sender
> You have not previously corresponded with this sender.
>
> ZjQcmQRYFpfptBannerEnd
> Hi David,
> these two metrics have two different objectives.
>
> > kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec <
>
> ****
> Produce request rate per topic. Omitting 'topic=(...)' will yield the
> all-topic rate.
> *****
>
> while
>
> >kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce|FetchConsumer|FetchFollower}
> <
>
> it is going to report as well requests per sec, however, you can filter
> and select by Produce/Fetch from consumers or Fetch from Followers.
> Note this is a metric collected at the network type, while the previous is
> at the server level.
>
> The first one will provide you with info per topic, which is a very
> valuable asset in order to know more about what kind of usage clients are
> doing from a topic perspective. Keep in mind the second one, will only give
> you the global request per second.
>
> So to your questions:
>
> > Shouldn't both metrics indicate the same?
>
> They indicate similar data at different levels, so they should not be far
> away, however, some distance (but minimal) between the two could be
> expected as they are collected at two different points.
>
> > seems to be the accurate one matching the number of messages sent with
> no batching.
>
> In a good monitoring solution, you should have both metrics as one will
> give you the overall load and the other the topic partition per broker view
> (aka what are your users doing).
>
> what I don't understand is the batching part of your question.
>
> If you like to see messages In I would suggest you to use
>
> > kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec <
> Aggregate incoming message rate.
>
> Sincerely
>
> -- Pere
>
>
> On Sat, Feb 18, 2023 at 12:17 AM David Ballano Fernandez <
> dfernan...@demonware.net> wrote:
>
>> Hi guys,
>>
>> I am having some confusion around 2 Kafka metrics:
>>
>> *Request rate.*
>>
>> kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce|FetchConsumer|FetchFollower}
>>
>> *Produce request rate.*
>> kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec
>>
>>
>> https://docs.confluent.io/platform/current/kafka/monitoring.html#server-metrics
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.confluent.io_platform_current_kafka_monitoring.html-23server-2Dmetrics&d=DwMFaQ&c=qE8EibqjfXM-zBfebVhd4gtjNZbrDcrKYXvb1gt38s4&r=p-f3AJg4e4Uk20g_16kSyBtabT4JOB-1GIb23_CxD58&m=S_eduaJSLvK0WMz0OiEppjB-dWgGa16RxmcEYZGwTvMMPf6CruRKu8aCf1_lk1L4&s=wHj2BcMsO0BRKhP_R042bVe3Vw9nhHFFWpB00gv2uVw&e=>
>>
>> They seem to be tracking the same, but when I graph them and say I pick
>> Producer. The numbers are different.
>>
>> and looks like
>>
>> *Produce request rate.*
>> kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec
>>
>> seems to be the accurate one matching the number of messages sent with no
>> batching.
>>
>> Shouldn't both metrics indicate the same?
>>
>> Thanks!
>>
>
>
> --
> Pere Urbon-Bayes
> Software Architect
> https://twitter.com/purbon
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_purbon&d=DwMFaQ&c=qE8EibqjfXM-zBfebVhd4gtjNZbrDcrKYXvb1gt38s4&r=p-f3AJg4e4Uk20g_16kSyBtabT4JOB-1GIb23_CxD58&m=S_eduaJSLvK0WMz0OiEppjB-dWgGa16RxmcEYZGwTvMMPf6CruRKu8aCf1_lk1L4&s=IeLld3jaDO53mdNUhmAdKzsjuxNI5z466ilSuvUOaFA&e=>
> https://www.linkedin.com/in/purbon/
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_in_purbon_&d=DwMFaQ&c=qE8EibqjfXM-zBfebVhd4gtjNZbrDcrKYXvb1gt38s4&r=p-f3AJg4e4Uk20g_16kSyBtabT4JOB-1GIb23_CxD58&m=S_eduaJSLvK0WMz0OiEppjB-dWgGa16RxmcEYZGwTvMMPf6CruRKu8aCf1_lk1L4&s=KepVYe2cAVyV3SU0NUn2g5rLyuY6QWmunVA5xXUWDVE&e=>
>
