[
https://issues.apache.org/jira/browse/KAFKA-9355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stanislav Savulchik updated KAFKA-9355:
---------------------------------------
Description:
*Steps to Reproduce*
Set processing.guarantee = exactly_once and remove local state dir in order to
force state restoration from changelog topics.
*Expected Behavior*
There are registered MBeans like
kafka.streams:type=stream-state-metrics,client-id=<application.id>-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1,task-id=0_0,rocksdb-state-id=<state.store>
for persistent RocksDB KeyValueStore-s after streams task state restoration.
*Actual Behavior*
There are no registered MBeans like above after streams task state restoration.
*Details*
I managed to inject custom MetricsReporter in order to log metricChange and
metricRemoval calls. According to the logs at some point the missing metrics
are removed and never restored later. Here is an excerpt for number-open-files
metric:
{noformat}
16:33:40.403 DEBUG c.m.r.LoggingMetricsReporter - metricChange MetricName
[name=number-open-files, group=stream-state-metrics, description=Number of
currently open files,
tags={client-id=morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1,
task-id=0_0, rocksdb-state-id=buffered-event}]
16:33:40.403 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
Recorder for buffered-event] Adding metrics recorder of task 0_0 to metrics
recording trigger
16:33:40.403 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
Recorder for buffered-event] Adding statistics for store buffered-event of task
0_0
16:33:40.610 INFO o.a.k.s.p.i.StoreChangelogReader - stream-thread
[morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1] No
checkpoint found for task 0_0 state store buffered-event changelog
morpheus.conversion-buffered-event-changelog-0 with EOS turned on.
Reinitializing the task and restore its state from the beginning.
16:33:40.610 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
Recorder for buffered-event] Removing statistics for store buffered-event of
task 0_0
16:33:40.610 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
Recorder for buffered-event] Removing metrics recorder for store buffered-event
of task 0_0 from metrics recording trigger
16:33
16:33:40.611 DEBUG c.m.r.LoggingMetricsReporter - metricRemoval MetricName
[name=number-open-files, group=stream-state-metrics, description=Number of
currently open files,
tags={client-id=morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1,
task-id=0_0, rocksdb-state-id=buffered-event}]
16:33:40.625 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
Recorder for buffered-event] Adding metrics recorder of task 0_0 to metrics
recording trigger
16:33:40.625 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
Recorder for buffered-event] Adding statistics for store buffered-event of task
0_0
...
(no more calls to metricChange for the removed number-open-files
metric){noformat}
Also a complete log is attached [^metric-removal.log]
was:
*Steps to Reproduce*
Set processing.guarantee = exactly_once and remove local state dir in order to
force state restoration from changelog topics.
*Expected Behavior*
There are registered MBeans like
kafka.streams:type=stream-state-metrics,client-id=<application.id>-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1,task-id=0_0,rocksdb-state-id=<state.store>
for persistent RockDB KeyValueStore-s after streams task state restoration.
*Actual Behavior*
There are no registered MBeans like above after streams task state restoration.
*Details*
I managed to inject custom MetricsReporter in order to log metricChange and
metricRemoval calls. According to the logs at some point the missing metrics
are removed and never restored later. Here is an excerpt for number-open-files
metric:
{noformat}
16:33:40.403 DEBUG c.m.r.LoggingMetricsReporter - metricChange MetricName
[name=number-open-files, group=stream-state-metrics, description=Number of
currently open files,
tags={client-id=morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1,
task-id=0_0, rocksdb-state-id=buffered-event}]
16:33:40.403 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
Recorder for buffered-event] Adding metrics recorder of task 0_0 to metrics
recording trigger
16:33:40.403 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
Recorder for buffered-event] Adding statistics for store buffered-event of task
0_0
16:33:40.610 INFO o.a.k.s.p.i.StoreChangelogReader - stream-thread
[morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1] No
checkpoint found for task 0_0 state store buffered-event changelog
morpheus.conversion-buffered-event-changelog-0 with EOS turned on.
Reinitializing the task and restore its state from the beginning.
16:33:40.610 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
Recorder for buffered-event] Removing statistics for store buffered-event of
task 0_0
16:33:40.610 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
Recorder for buffered-event] Removing metrics recorder for store buffered-event
of task 0_0 from metrics recording trigger
16:33
16:33:40.611 DEBUG c.m.r.LoggingMetricsReporter - metricRemoval MetricName
[name=number-open-files, group=stream-state-metrics, description=Number of
currently open files,
tags={client-id=morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1,
task-id=0_0, rocksdb-state-id=buffered-event}]
16:33:40.625 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
Recorder for buffered-event] Adding metrics recorder of task 0_0 to metrics
recording trigger
16:33:40.625 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
Recorder for buffered-event] Adding statistics for store buffered-event of task
0_0
...
(no more calls to metricChange for the removed number-open-files
metric){noformat}
Also a complete log is attached [^metric-removal.log]
> RocksDB statistics are removed from JMX when EOS enabled and empty local
> state dir
> ----------------------------------------------------------------------------------
>
> Key: KAFKA-9355
> URL: https://issues.apache.org/jira/browse/KAFKA-9355
> Project: Kafka
> Issue Type: Bug
> Components: metrics, streams
> Affects Versions: 2.4.0
> Reporter: Stanislav Savulchik
> Priority: Major
> Attachments: metric-removal.log
>
>
> *Steps to Reproduce*
> Set processing.guarantee = exactly_once and remove local state dir in order
> to force state restoration from changelog topics.
> *Expected Behavior*
> There are registered MBeans like
> kafka.streams:type=stream-state-metrics,client-id=<application.id>-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1,task-id=0_0,rocksdb-state-id=<state.store>
> for persistent RocksDB KeyValueStore-s after streams task state restoration.
> *Actual Behavior*
> There are no registered MBeans like above after streams task state
> restoration.
> *Details*
> I managed to inject custom MetricsReporter in order to log metricChange and
> metricRemoval calls. According to the logs at some point the missing metrics
> are removed and never restored later. Here is an excerpt for
> number-open-files metric:
> {noformat}
> 16:33:40.403 DEBUG c.m.r.LoggingMetricsReporter - metricChange MetricName
> [name=number-open-files, group=stream-state-metrics, description=Number of
> currently open files,
> tags={client-id=morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1,
> task-id=0_0, rocksdb-state-id=buffered-event}]
> 16:33:40.403 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Adding metrics recorder of task 0_0 to metrics
> recording trigger
> 16:33:40.403 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Adding statistics for store buffered-event of
> task 0_0
> 16:33:40.610 INFO o.a.k.s.p.i.StoreChangelogReader - stream-thread
> [morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1] No
> checkpoint found for task 0_0 state store buffered-event changelog
> morpheus.conversion-buffered-event-changelog-0 with EOS turned on.
> Reinitializing the task and restore its state from the beginning.
> 16:33:40.610 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Removing statistics for store buffered-event of
> task 0_0
> 16:33:40.610 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Removing metrics recorder for store
> buffered-event of task 0_0 from metrics recording trigger
> 16:33
> 16:33:40.611 DEBUG c.m.r.LoggingMetricsReporter - metricRemoval MetricName
> [name=number-open-files, group=stream-state-metrics, description=Number of
> currently open files,
> tags={client-id=morpheus.conversion-678e4b25-8fc7-4266-85a0-7a5fe52a4060-StreamThread-1,
> task-id=0_0, rocksdb-state-id=buffered-event}]
> 16:33:40.625 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Adding metrics recorder of task 0_0 to metrics
> recording trigger
> 16:33:40.625 DEBUG o.a.k.s.s.i.m.RocksDBMetricsRecorder - [RocksDB Metrics
> Recorder for buffered-event] Adding statistics for store buffered-event of
> task 0_0
> ...
> (no more calls to metricChange for the removed number-open-files
> metric){noformat}
> Also a complete log is attached [^metric-removal.log]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
