Hi, Magnus,
Thanks for the updated KIP. Just a couple of more comments.
50. To troubleshoot a particular client issue, I imagine that the client
needs to identify its client_instance_id. How does the client find this
out? Do we plan to include client_instance_id in the client log, expose it
as a metric or something else?
51. The KIP lists a bunch of metrics that need to be collected at the
client side. However, it seems quite a few useful java client metrics like
the following are missing.
buffer-total-bytes
buffer-available-bytes
bufferpool-wait-time
batch-size-avg
batch-size-max
io-wait-ratio
io-ratio
Thanks,
Jun
On Mon, Apr 4, 2022 at 10:01 AM Jun Rao <j...@confluent.io> wrote:
> Hi, Xavier,
>
> Thanks for the reply.
>
> 28. It does seem that we have started using KafkaMetrics on the broker
> side. Then, my only concern is on the usage of Histogram in KafkaMetrics.
> Histogram in KafkaMetrics statically divides the value space into a fixed
> number of buckets and only returns values on the bucket boundary. So, the
> returned histogram value may never show up in a recorded value. Yammer
> Histogram, on the other hand, uses reservoir sampling. The reported value
> is always one of the recorded values. So, I am not sure that Histogram in
> KafkaMetrics is as good as Yammer Histogram. ClientMetricsPluginExportTime
> uses Histogram.
>
> Thanks,
>
> Jun
>
> On Thu, Mar 31, 2022 at 5:21 PM Xavier Léauté <xav...@confluent.io.invalid>
> wrote:
>
>> >
>> > 28. On the broker, we typically use Yammer metrics. Only for metrics
>> that
>> > depend on Kafka metric features (e.g., quota), we use the Kafka metric.
>> > Yammer metrics have 4 types: gauge, meter, histogram and timer. meter
>> > calculates a rate, but also exposes an accumulated value.
>> >
>>
>> I don't see a good reason we should limit ourselves to Yammer metrics on
>> the broker. KafkaMetrics was written
>> to replace Yammer metrics and is used for all new components (clients,
>> streams, connect, etc.)
>> My understanding is that the original goal was to retire Yammer metrics in
>> the broker in favor of KafkaMetrics.
>> We just haven't done so out of backwards compatibility concerns.
>> There are other broker metrics such as group coordinator, transaction
>> state
>> manager, and various socket server metrics
>> already using KafkaMetrics that don't need specific Kafka metric features,
>> so I don't see why we should refrain from using
>> Kafka metrics on the broker unless there are real compatibility concerns
>> or
>> where implementation specifics could lead to confusion when comparing
>> metrics using different implementations.
>>
>> In my opinion we should encourage people to use KafkaMetrics going forward
>> on the broker as well, for two reasons:
>> a) yammer metrics is long deprecated and no longer maintained
>> b) yammer metrics are much less expressive
>> c) we don't have a proper API to expose yammer metrics outside of JMX
>> (MetricsReporter only exposes KafkaMetrics)
>>
>
