Hi there, Recently I performed several rolling upgrades following official steps for our Kafka brokers from 0.11.0.1 to newer versions in different environments, and apparently working fine in all cases from functional point of view: producers and consumers working as expected.
Specifically, I upgraded: 1. From 0.11.0.1 to 1.0.0, and then from 1.0.0 to 2.0.0, and then to 2.1.1 2. From 0.11.0.1 directly to 2.1.1 However, in all cases JMX metrics for Fetch and Produce which used to show all producers and consumers working with brokers are gone, just showing queue-size, in our JMX monitoring clients -specifically, Introscope Wily- keeping same configuration. In fact, I removed Wily filter configuration for JMX in order to show all possible metrics, and keeping both Fetch and Produce still gone. Note I checked if having proper version after rolling upgrade, for example, for 2.1.1, and being as expected: ll /opt/kafka/libs/ total 54032 -rw-r--r-- 1 kafka kafka 69409 Jan 4 08:42 activation-1.1.1.jar -rw-r--r-- 1 kafka kafka 14768 Jan 4 08:42 aopalliance-repackaged-2.5.0-b42.jar -rw-r--r-- 1 kafka kafka 90347 Jan 4 08:42 argparse4j-0.7.0.jar -rw-r--r-- 1 kafka kafka 20437 Jan 4 08:40 audience-annotations-0.5.0.jar -rw-r--r-- 1 kafka kafka 501879 Jan 4 08:43 commons-lang3-3.8.1.jar -rw-r--r-- 1 kafka kafka 96801 Feb 8 18:32 connect-api-2.1.1.jar -rw-r--r-- 1 kafka kafka 18265 Feb 8 18:32 connect-basic-auth-extension-2.1.1.jar -rw-r--r-- 1 kafka kafka 20509 Feb 8 18:32 connect-file-2.1.1.jar -rw-r--r-- 1 kafka kafka 45489 Feb 8 18:32 connect-json-2.1.1.jar -rw-r--r-- 1 kafka kafka 466588 Feb 8 18:32 connect-runtime-2.1.1.jar -rw-r--r-- 1 kafka kafka 90358 Feb 8 18:32 connect-transforms-2.1.1.jar -rw-r--r-- 1 kafka kafka 2442625 Jan 4 08:43 guava-20.0.jar -rw-r--r-- 1 kafka kafka 186763 Jan 4 08:42 hk2-api-2.5.0-b42.jar -rw-r--r-- 1 kafka kafka 189454 Jan 4 08:42 hk2-locator-2.5.0-b42.jar -rw-r--r-- 1 kafka kafka 135317 Jan 4 08:42 hk2-utils-2.5.0-b42.jar -rw-r--r-- 1 kafka kafka 66894 Jan 11 21:28 jackson-annotations-2.9.8.jar -rw-r--r-- 1 kafka kafka 325619 Jan 11 21:27 jackson-core-2.9.8.jar -rw-r--r-- 1 kafka kafka 1347236 Jan 11 21:27 jackson-databind-2.9.8.jar -rw-r--r-- 1 kafka kafka 32373 Jan 11 21:28 jackson-jaxrs-base-2.9.8.jar -rw-r--r-- 1 kafka kafka 15861 Jan 11 21:28 jackson-jaxrs-json-provider-2.9.8.jar -rw-r--r-- 1 kafka kafka 32627 Jan 11 21:28 jackson-module-jaxb-annotations-2.9.8.jar -rw-r--r-- 1 kafka kafka 737884 Jan 4 08:43 javassist-3.22.0-CR2.jar -rw-r--r-- 1 kafka kafka 26366 Jan 4 08:42 javax.annotation-api-1.2.jar -rw-r--r-- 1 kafka kafka 2497 Jan 4 08:42 javax.inject-1.jar -rw-r--r-- 1 kafka kafka 5951 Jan 4 08:42 javax.inject-2.5.0-b42.jar -rw-r--r-- 1 kafka kafka 95806 Jan 4 08:42 javax.servlet-api-3.1.0.jar -rw-r--r-- 1 kafka kafka 126898 Jan 4 08:42 javax.ws.rs-api-2.1.1.jar -rw-r--r-- 1 kafka kafka 127509 Jan 4 08:42 javax.ws.rs-api-2.1.jar -rw-r--r-- 1 kafka kafka 125632 Jan 4 08:42 jaxb-api-2.3.0.jar -rw-r--r-- 1 kafka kafka 181563 Jan 4 08:42 jersey-client-2.27.jar -rw-r--r-- 1 kafka kafka 1140395 Jan 4 08:43 jersey-common-2.27.jar -rw-r--r-- 1 kafka kafka 18085 Jan 4 08:42 jersey-container-servlet-2.27.jar -rw-r--r-- 1 kafka kafka 59332 Jan 4 08:42 jersey-container-servlet-core-2.27.jar -rw-r--r-- 1 kafka kafka 62547 Jan 4 08:42 jersey-hk2-2.27.jar -rw-r--r-- 1 kafka kafka 71936 Jan 4 08:42 jersey-media-jaxb-2.27.jar -rw-r--r-- 1 kafka kafka 933619 Jan 4 08:43 jersey-server-2.27.jar -rw-r--r-- 1 kafka kafka 280573 Jan 4 08:43 jetty-client-9.4.12.v20180830.jar -rw-r--r-- 1 kafka kafka 16680 Jan 4 08:42 jetty-continuation-9.4.12.v20180830.jar -rw-r--r-- 1 kafka kafka 195664 Jan 4 08:42 jetty-http-9.4.12.v20180830.jar -rw-r--r-- 1 kafka kafka 139436 Jan 4 08:42 jetty-io-9.4.12.v20180830.jar -rw-r--r-- 1 kafka kafka 93077 Jan 4 08:42 jetty-security-9.4.12.v20180830.jar -rw-r--r-- 1 kafka kafka 609290 Jan 4 08:43 jetty-server-9.4.12.v20180830.jar -rw-r--r-- 1 kafka kafka 110983 Jan 4 08:42 jetty-servlet-9.4.12.v20180830.jar -rw-r--r-- 1 kafka kafka 93161 Jan 4 08:42 jetty-servlets-9.4.12.v20180830.jar -rw-r--r-- 1 kafka kafka 502985 Jan 4 08:43 jetty-util-9.4.12.v20180830.jar -rw-r--r-- 1 kafka kafka 78146 Jan 4 08:40 jopt-simple-5.0.4.jar -rw-r--r-- 1 kafka kafka 3949187 Feb 8 18:32 kafka_2.12-2.1.1.jar -rw-r--r-- 1 kafka kafka 821 Feb 8 18:32 kafka_2.12-2.1.1.jar.asc -rw-r--r-- 1 kafka kafka 31034 Feb 8 18:32 kafka_2.12-2.1.1-javadoc.jar -rw-r--r-- 1 kafka kafka 821 Feb 8 18:32 kafka_2.12-2.1.1-javadoc.jar.asc -rw-r--r-- 1 kafka kafka 10261 Feb 8 18:32 kafka_2.12-2.1.1-scaladoc.jar -rw-r--r-- 1 kafka kafka 821 Feb 8 18:32 kafka_2.12-2.1.1-scaladoc.jar.asc -rw-r--r-- 1 kafka kafka 728679 Feb 8 18:32 kafka_2.12-2.1.1-sources.jar -rw-r--r-- 1 kafka kafka 821 Feb 8 18:32 kafka_2.12-2.1.1-sources.jar.asc -rw-r--r-- 1 kafka kafka 2956429 Feb 8 18:32 kafka_2.12-2.1.1-test.jar -rw-r--r-- 1 kafka kafka 821 Feb 8 18:32 kafka_2.12-2.1.1-test.jar.asc -rw-r--r-- 1 kafka kafka 740522 Feb 8 18:32 kafka_2.12-2.1.1-test-sources.jar -rw-r--r-- 1 kafka kafka 821 Feb 8 18:32 kafka_2.12-2.1.1-test-sources.jar.asc -rw-r--r-- 1 kafka kafka 1893108 Feb 8 18:31 kafka-clients-2.1.1.jar -rw-r--r-- 1 kafka kafka 14499 Feb 8 18:32 kafka-log4j-appender-2.1.1.jar -rw-r--r-- 1 kafka kafka 871817 Feb 8 18:32 kafka-streams-2.1.1.jar -rw-r--r-- 1 kafka kafka 39331 Feb 8 18:32 kafka-streams-examples-2.1.1.jar -rw-r--r-- 1 kafka kafka 170375 Feb 8 18:33 kafka-streams-scala_2.12-2.1.1.jar -rw-r--r-- 1 kafka kafka 38309 Feb 8 18:32 kafka-streams-test-utils-2.1.1.jar -rw-r--r-- 1 kafka kafka 328053 Feb 8 18:32 kafka-tools-2.1.1.jar -rw-r--r-- 1 kafka kafka 489884 Jan 4 08:39 log4j-1.2.17.jar -rw-r--r-- 1 kafka kafka 511547 Jan 4 08:39 lz4-java-1.5.0.jar -rw-r--r-- 1 kafka kafka 55051 Jan 4 08:43 maven-artifact-3.6.0.jar -rw-r--r-- 1 kafka kafka 82123 Jan 4 08:40 metrics-core-2.2.0.jar -rw-r--r-- 1 kafka kafka 20235 Jan 4 08:42 osgi-resource-locator-1.0.1.jar -rw-r--r-- 1 kafka kafka 261617 Jan 4 08:43 plexus-utils-3.1.0.jar -rw-r--r-- 1 kafka kafka 130999 Jan 4 08:43 reflections-0.9.11.jar -rw-r--r-- 1 kafka kafka 13908249 Jan 7 19:26 rocksdbjni-5.14.2.jar -rw-r--r-- 1 kafka kafka 5277511 Jan 4 08:40 scala-library-2.12.7.jar -rw-r--r-- 1 kafka kafka 54709 Jan 4 08:40 scala-logging_2.12-3.9.0.jar -rw-r--r-- 1 kafka kafka 3616372 Jan 4 08:40 scala-reflect-2.12.7.jar -rw-r--r-- 1 kafka kafka 41203 Jan 4 08:36 slf4j-api-1.7.25.jar -rw-r--r-- 1 kafka kafka 12244 Jan 4 08:39 slf4j-log4j12-1.7.25.jar -rw-r--r-- 1 kafka kafka 2018686 Jan 4 08:39 snappy-java-1.1.7.2.jar -rw-r--r-- 1 kafka kafka 63777 Jan 4 08:42 validation-api-1.1.0.Final.jar -rw-r--r-- 1 kafka kafka 74626 Jan 4 08:40 zkclient-0.11.jar -rw-r--r-- 1 kafka kafka 906708 Jan 4 08:40 zookeeper-3.4.13.jar -rw-r--r-- 1 kafka kafka 3810541 Jan 4 08:39 zstd-jni-1.3.7-1.jar Any idea why is this happening? Maybe upgrade was wrong? Maybe related to any broker missconfiguration? Thanks [https://www.williamhillplc.com/content/signature/WHlogo.gif?width=180]<http://www.williamhill.com/> [https://www.williamhillplc.com/content/signature/senet.gif?width=180]<http://www.whenthefunstops.co.uk/> Jose Manuel Vega Monroy Java Developer / Software Developer Engineer in Test Direct: +0035 0 2008038 (Ext. 8038) Email: jose.mon...@williamhill.com<mailto:jose.mon...@williamhill.com> William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
