Hi Peter, Are you running the Prometheus exporter on a broker? Or on a Kafka client?
Also, what version? Cheers, Liam Clarke-Hutchinson On Wed, 6 Apr 2022 at 19:14, Peter Schrott (Extern) <peter.schr...@mapp.com.invalid> wrote: > Hi users, > > I am getting this following exceptions occasionally: > JMX scrape failed: javax.management.RuntimeMBeanException: getMBeanInfo > threw RuntimeException > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1384) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:920) > at io.prometheus.jmx.shaded.io > .prometheus.jmx.JmxScraper.scrapeBean(JmxScraper.java:130) > at io.prometheus.jmx.shaded.io > .prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:117) > at io.prometheus.jmx.shaded.io > .prometheus.jmx.JmxCollector.collect(JmxCollector.java:542) > at io.prometheus.jmx.shaded.io > .prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:190) > at io.prometheus.jmx.shaded.io > .prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:223) > at io.prometheus.jmx.shaded.io > .prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:144) > at io.prometheus.jmx.shaded.io > .prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22) > at io.prometheus.jmx.shaded.io > .prometheus.client.exporter.HTTPServer$HTTPMetricHandler.handle(HTTPServer.java:68) > at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79) > at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83) > at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82) > at > sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675) > at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79) > at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.util.ConcurrentModificationException > at java.util.HashMap$HashIterator.nextNode(HashMap.java:1445) > at java.util.HashMap$EntryIterator.next(HashMap.java:1479) > at java.util.HashMap$EntryIterator.next(HashMap.java:1477) > at > org.apache.kafka.common.metrics.JmxReporter$KafkaMbean.getMBeanInfo(JmxReporter.java:279) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1378) > ... 18 more > > Looking at the code here: > > https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/metrics/JmxReporter.java#L279 > it is obviously happening while looping over the private final Map<String, > KafkaMetric> metrics. The metrics attribute is not „locked“ or of type > ConcurrentHashMap. > Prometheus JMX Exporter tries to scrape the Beans while (I assume) Kafka > Consumer is updating them. > > Am I getting something wrong or am I holding it wrong? Anyone an idea? > > Thanks & Best Peter > > > Mapp Digital Germany GmbH with registered offices at Dachauer, Str. 63, > 80335 München. > Registered with the District Court München HRB 226181 > Managing Directors: Frasier, Christopher & Warren, Steve > This e-mail is from Mapp Digital and its international legal entities and > may contain information that is confidential or proprietary. > If you are not the intended recipient, do not read, copy or distribute the > e-mail or any attachments. Instead, please notify the sender and delete the > e-mail and any attachments. > Please consider the environment before printing. Thank you. >
