It looks like there is indeed a bug in kafka-streams 1.1.0. I think what was happening was the time spent processing each record in ns was being added to the total metric instead of incrementing by 1 for each record. Looks like the implementation has been changed in trunk. I don't see any commit messages mentioning this particular issue, but hopefully the change fixes it.
________________________________ From: Sam Lendle Sent: Wednesday, June 20, 2018 6:10:03 PM To: users@kafka.apache.org Subject: Some Total and Rate metrics are not consistent I’m trying to use the total metrics introduced in KIP-187 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-187+-+Add+cumulative+count+metric+for+all+Kafka+rate+metrics) For some metrics, the total and rates are not consistent. In particular, for stream-processor-node-metrics, I’m seeing about 500-800 operations per second in a particular streams thread/processor node as reported by the process-rate metric, but the process-total metric is increasing by about 100 million per second. See attached screenshot from VisualVM. Other metrics seem fine, for example forward-rate and forward-total metrics under stream-processor-node-metrics are consistent. Am I misunderstanding the interpretation of the –total metrics? If this is a bug, can I do anything in addition to this email to report it? File a JIRA? Best, Sam [cid:1c705c18-80bd-4fbf-8447-86d6d6527ea0]
