Steven Schlansker created KAFKA-21078:
-----------------------------------------
Summary: Telemetry reporter pushes metrics whose sensors are
disabled by metrics.recording.level
Key: KAFKA-21078
URL: https://issues.apache.org/jira/browse/KAFKA-21078
Project: Kafka
Issue Type: Improvement
Components: clients, streams
Affects Versions: 4.3.1
Reporter: Steven Schlansker
{\{Sensor.add}} registers every metric with the registry and thus with every
\{{MetricsReporter}}, regardless of whether the sensor's \{{RecordingLevel}} is
enabled by \{{metrics.recording.level}}. Only \{{Sensor.record}} checks
\{{shouldRecord}}. As a result, an application at INFO still exposes every
DEBUG-level metric with a value of 0 (a \{{Rate}} with no samples measures 0,
not NaN).
{\{KafkaMetricsCollector.collectMetric}} has two filters: the subscription
prefix (\{{MetricsEmitter.shouldEmitMetric}}) and "value is a Number". It has
no notion of recording level and does not drop zero values. So the KIP-714 push
carries every disabled metric with its full label set on every interval, and in
delta mode it emits a fresh zero delta for each disabled sum.
For Kafka Streams (KIP-1076) the disabled DEBUG metrics are most of the
payload: per-store per-operation rates (\{{stream-state-metrics}}), per-node
\{{process-rate}}/\{{process-total}}, and task-level \{{process-\*}}. In one
production topology at INFO, \{{org.apache.kafka.stream.state.\*.rate}} alone
was several thousand series per stream thread, all zero. This inflates the push
toward \{{telemetry.max.bytes}} (see the linked bug) and the cardinality of the
receiving backend.
{\{KafkaMetric}} does not know its sensor's level
(\{{MetricConfig.recordLevel()}} is the configured threshold, not the sensor's
own level), so the collector cannot filter today. Options:
* \{{Sensor.add}} skips \{{registry.registerMetric}} when \{{shouldRecord()}}
is false. The level is fixed at construction, so nothing can later enable the
sensor. This also cleans up JMX.
* \{{Metrics}} exposes the sensor level for a metric, and
\{{KafkaMetricsCollector}} skips metrics whose sensor is disabled.
The first option changes what JMX shows and needs a KIP note. The second is
local to the telemetry reporter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
