RuntimeMBeanException: getMBeanInfo threw RuntimeException caused by ConcurrentModificationException
Hi users, I am getting this following exceptions occasionally: JMX scrape failed: javax.management.RuntimeMBeanException: getMBeanInfo threw RuntimeException at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1384) at com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:920) at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.scrapeBean(JmxScraper.java:130) at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:117) at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:542) at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:190) at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:223) at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:144) at io.prometheus.jmx.shaded.io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22) at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer$HTTPMetricHandler.handle(HTTPServer.java:68) at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79) at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83) at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82) at sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675) at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79) at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextNode(HashMap.java:1445) at java.util.HashMap$EntryIterator.next(HashMap.java:1479) at java.util.HashMap$EntryIterator.next(HashMap.java:1477) at org.apache.kafka.common.metrics.JmxReporter$KafkaMbean.getMBeanInfo(JmxReporter.java:279) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1378) ... 18 more Looking at the code here: https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/metrics/JmxReporter.java#L279 it is obviously happening while looping over the private final Map metrics. The metrics attribute is not „locked“ or of type ConcurrentHashMap. Prometheus JMX Exporter tries to scrape the Beans while (I assume) Kafka Consumer is updating them. Am I getting something wrong or am I holding it wrong? Anyone an idea? Thanks & Best Peter Mapp Digital Germany GmbH with registered offices at Dachauer, Str. 63, 80335 München. Registered with the District Court München HRB 226181 Managing Directors: Frasier, Christopher & Warren, Steve This e-mail is from Mapp Digital and its international legal entities and may contain information that is confidential or proprietary. If you are not the intended recipient, do not read, copy or distribute the e-mail or any attachments. Instead, please notify the sender and delete the e-mail and any attachments. Please consider the environment before printing. Thank you.
Fwd: RuntimeMBeanException: getMBeanInfo threw RuntimeException caused by ConcurrentModificationException
Hi Peter, Are you running the Prometheus exporter on a broker? Or on a Kafka client? Also, what version? Cheers, Liam Clarke-Hutchinson On Wed, 6 Apr 2022 at 19:14, Peter Schrott (Extern) wrote: > Hi users, > > I am getting this following exceptions occasionally: > JMX scrape failed: javax.management.RuntimeMBeanException: getMBeanInfo > threw RuntimeException > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1384) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:920) > at io.prometheus.jmx.shaded.io > .prometheus.jmx.JmxScraper.scrapeBean(JmxScraper.java:130) > at io.prometheus.jmx.shaded.io > .prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:117) > at io.prometheus.jmx.shaded.io > .prometheus.jmx.JmxCollector.collect(JmxCollector.java:542) > at io.prometheus.jmx.shaded.io > .prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:190) > at io.prometheus.jmx.shaded.io > .prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:223) > at io.prometheus.jmx.shaded.io > .prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:144) > at io.prometheus.jmx.shaded.io > .prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22) > at io.prometheus.jmx.shaded.io > .prometheus.client.exporter.HTTPServer$HTTPMetricHandler.handle(HTTPServer.java:68) > at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79) > at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83) > at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82) > at > sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675) > at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79) > at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.util.ConcurrentModificationException > at java.util.HashMap$HashIterator.nextNode(HashMap.java:1445) > at java.util.HashMap$EntryIterator.next(HashMap.java:1479) > at java.util.HashMap$EntryIterator.next(HashMap.java:1477) > at > org.apache.kafka.common.metrics.JmxReporter$KafkaMbean.getMBeanInfo(JmxReporter.java:279) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1378) > ... 18 more > > Looking at the code here: > > https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/metrics/JmxReporter.java#L279 > it is obviously happening while looping over the private final Map KafkaMetric> metrics. The metrics attribute is not „locked“ or of type > ConcurrentHashMap. > Prometheus JMX Exporter tries to scrape the Beans while (I assume) Kafka > Consumer is updating them. > > Am I getting something wrong or am I holding it wrong? Anyone an idea? > > Thanks & Best Peter > > > Mapp Digital Germany GmbH with registered offices at Dachauer, Str. 63, > 80335 München. > Registered with the District Court München HRB 226181 > Managing Directors: Frasier, Christopher & Warren, Steve > This e-mail is from Mapp Digital and its international legal entities and > may contain information that is confidential or proprietary. > If you are not the intended recipient, do not read, copy or distribute the > e-mail or any attachments. Instead, please notify the sender and delete the > e-mail and any attachments. > Please consider the environment before printing. Thank you. >
Move all partitions from one log.dir to another
Hey 👋 I am using a jbod setup in a 2.8 Kafka cluster. I started with only one disk in my JBOD, all partitions are on one volume (one log.dir) I have added a disk with the right log.dir and the brokers are well configured, I would like to move all replicas of all partitions without exception from the first volume to the new one. With the `kafka-reassign-partitions` command it seems to be a bit too much trouble. I need to generate a `Proposed partition reassignment configuration` and then modify the paths to the log_dirs dynamically according to what is in the `replicas` list. It can be automated but I wonder if there is a simpler solution for my relatively simple need. Thanks
Java 8 support in Kafka 3.1.1 release
Hi Luke, Our application uses Java 8 but it seems the java 8 support is deprecated from Kafka 3.0.0 release onwards. Please let us know can we still use Java 8 in Kafka upcoming release 3.1.1? Regards, Deepak
Re: Java 8 support in Kafka 3.1.1 release
Hi Deepak, Yes, you can continue to use Java 8 until Kafka v4.0 released. You can check the KIP-750 for more detail: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308223 Thank you. Luke On Thu, Apr 7, 2022 at 3:05 AM Deepak Jain wrote: > Hi Luke, > > > > Our application uses Java 8 but it seems the java 8 support is deprecated > from Kafka 3.0.0 release onwards. Please let us know can we still use Java > 8 in Kafka upcoming release 3.1.1? > > > > Regards, > > Deepak >
Re: Move all partitions from one log.dir to another
Hi Fares, I don't know if there is other simpler solution, but I think the `kafka-reassign-partitions` command is the safest way. Thank you. Luke On Wed, Apr 6, 2022 at 11:32 PM Fares Oueslati wrote: > Hey 👋 > I am using a jbod setup in a 2.8 Kafka cluster. > > I started with only one disk in my JBOD, all partitions are on one volume > (one log.dir) > > I have added a disk with the right log.dir and the brokers are well > configured, I would like to move all replicas of all partitions without > exception from the first volume to the new one. > > With the `kafka-reassign-partitions` command it seems to be a bit too much > trouble. > I need to generate a `Proposed partition reassignment configuration` and > then modify the paths to the log_dirs dynamically according to what is in > the `replicas` list. > > It can be automated but I wonder if there is a simpler solution for my > relatively simple need. > > Thanks >
