Hi, This is exception is happening on a Kafka client - a consumer.
The Kafka client is of version 2.8.1 (same as the broker). The jmx exporter is running as java agent in version 0.14.0. Thanks, Peter > Am 06.04.2022 um 14:47 schrieb Liam Clarke-Hutchinson <lclar...@redhat.com>: > > > Hi Peter, > > Are you running the Prometheus exporter on a broker? Or on a Kafka client? > > Also, what version? > > Cheers, > > Liam Clarke-Hutchinson > > On Wed, 6 Apr 2022 at 19:14, Peter Schrott (Extern) > <peter.schr...@mapp.com.invalid> wrote: > >> Hi users, >> >> I am getting this following exceptions occasionally: >> JMX scrape failed: javax.management.RuntimeMBeanException: getMBeanInfo >> threw RuntimeException >> at >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1384) >> at >> com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:920) >> at io.prometheus.jmx.shaded.io >> .prometheus.jmx.JmxScraper.scrapeBean(JmxScraper.java:130) >> at io.prometheus.jmx.shaded.io >> .prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:117) >> at io.prometheus.jmx.shaded.io >> .prometheus.jmx.JmxCollector.collect(JmxCollector.java:542) >> at io.prometheus.jmx.shaded.io >> .prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:190) >> at io.prometheus.jmx.shaded.io >> .prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:223) >> at io.prometheus.jmx.shaded.io >> .prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:144) >> at io.prometheus.jmx.shaded.io >> .prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22) >> at io.prometheus.jmx.shaded.io >> .prometheus.client.exporter.HTTPServer$HTTPMetricHandler.handle(HTTPServer.java:68) >> at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79) >> at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83) >> at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82) >> at >> sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675) >> at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79) >> at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> at java.lang.Thread.run(Thread.java:748) >> Caused by: java.util.ConcurrentModificationException >> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1445) >> at java.util.HashMap$EntryIterator.next(HashMap.java:1479) >> at java.util.HashMap$EntryIterator.next(HashMap.java:1477) >> at >> org.apache.kafka.common.metrics.JmxReporter$KafkaMbean.getMBeanInfo(JmxReporter.java:279) >> at >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1378) >> ... 18 more >> >> Looking at the code here: >> >> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/metrics/JmxReporter.java#L279 >> it is obviously happening while looping over the private final Map<String, >> KafkaMetric> metrics. The metrics attribute is not „locked“ or of type >> ConcurrentHashMap. >> Prometheus JMX Exporter tries to scrape the Beans while (I assume) Kafka >> Consumer is updating them. >> >> Am I getting something wrong or am I holding it wrong? Anyone an idea? >> >> Thanks & Best Peter >> >> >> Mapp Digital Germany GmbH with registered offices at Dachauer, Str. 63, >> 80335 München. >> Registered with the District Court München HRB 226181 >> Managing Directors: Frasier, Christopher & Warren, Steve >> This e-mail is from Mapp Digital and its international legal entities and >> may contain information that is confidential or proprietary. >> If you are not the intended recipient, do not read, copy or distribute the >> e-mail or any attachments. Instead, please notify the sender and delete the >> e-mail and any attachments. >> Please consider the environment before printing. Thank you. >> Mapp Digital Germany GmbH with registered offices at Dachauer, Str. 63, 80335 München. Registered with the District Court München HRB 226181 Managing Directors: Frasier, Christopher & Warren, Steve This e-mail is from Mapp Digital and its international legal entities and may contain information that is confidential or proprietary. If you are not the intended recipient, do not read, copy or distribute the e-mail or any attachments. Instead, please notify the sender and delete the e-mail and any attachments. Please consider the environment before printing. Thank you.
