I am able to reproduce the delay using 8.8.8.8. I'm not sure how the JVM's MBean server avoids this. Feel free to open a Jira to track this.
Justin On Thu, Jan 23, 2025 at 11:34 AM Vilius Šumskas <vilius.sums...@rivile.lt> wrote: > Can you try with -Djava.rmi.server.hostname=some.random.ip instead of > hostname? > > I tried with "foo" hostname and there is no delay, but switched to 8.8.8.8 > and delay is back again. Actually specifying any IP not under host's > interface produces the same result. > > -- > Vilius > > -----Original Message----- > From: Justin Bertram <jbert...@apache.org> > Sent: Thursday, January 23, 2025 5:58 PM > To: users@activemq.apache.org > Subject: Re: JMX exporter in Docker image > > From what I can tell the delay is shown in the first thread dump [1]: > > "main" #1 prio=5 os_prio=0 cpu=2709.38ms elapsed=33.95s > tid=0x00007f723002ae60 nid=0x33633 runnable [0x00007f7237121000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.Net.poll(java.base@17.0.13/Native Method) > at sun.nio.ch.NioSocketImpl.park(java.base@17.0.13 > /NioSocketImpl.java:186) > at sun.nio.ch.NioSocketImpl.park(java.base@17.0.13 > /NioSocketImpl.java:195) > at sun.nio.ch.NioSocketImpl.implRead(java.base@17.0.13 > /NioSocketImpl.java:319) > at sun.nio.ch.NioSocketImpl.read(java.base@17.0.13 > /NioSocketImpl.java:355) > at sun.nio.ch.NioSocketImpl$1.read(java.base@17.0.13 > /NioSocketImpl.java:808) > at java.net.Socket$SocketInputStream.read(java.base@17.0.13 > /Socket.java:966) > at java.io.BufferedInputStream.fill(java.base@17.0.13 > /BufferedInputStream.java:244) > at java.io.BufferedInputStream.read(java.base@17.0.13 > /BufferedInputStream.java:263) > - locked <0x000000009f631900> (a java.io.BufferedInputStream) > at java.io.DataInputStream.readUnsignedByte(java.base@17.0.13 > /DataInputStream.java:288) > at java.io.DataInputStream.readByte(java.base@17.0.13 > /DataInputStream.java:268) > at sun.rmi.transport.StreamRemoteCall.executeCall(java.rmi@17.0.13 > /StreamRemoteCall.java:241) > at sun.rmi.server.UnicastRef.invoke(java.rmi@17.0.13 > /UnicastRef.java:381) > at sun.rmi.registry.RegistryImpl_Stub.bind(java.rmi@17.0.13 > /RegistryImpl_Stub.java:73) > at > com.sun.jndi.rmi.registry.RegistryContext.bind(jdk.naming.rmi@17.0.13 > /RegistryContext.java:157) > at > com.sun.jndi.toolkit.url.GenericURLContext.bind(java.naming@17.0.13 > /GenericURLContext.java:243) > at javax.naming.InitialContext.bind(java.naming@17.0.13 > /InitialContext.java:417) > at > > javax.management.remote.rmi.RMIConnectorServer.bind(java.management.rmi@17.0.13 > /RMIConnectorServer.java:693) > at > > javax.management.remote.rmi.RMIConnectorServer.start(java.management.rmi@17.0.13 > /RMIConnectorServer.java:476) > - locked <0x000000009f7facf8> (a > javax.management.remote.rmi.RMIConnectorServer) > at > > org.apache.activemq.artemis.core.server.management.ConnectorServerFactory.init(ConnectorServerFactory.java:226) > at > > org.apache.activemq.artemis.core.server.management.ManagementConnector.start(ManagementConnector.java:96) > at > > org.apache.activemq.artemis.core.server.management.ManagementContext.start(ManagementContext.java:56) > - locked <0x00000000809db358> (a > org.apache.activemq.artemis.core.server.management.ManagementContext) > at > org.apache.activemq.artemis.cli.commands.Run$1.preActivate(Run.java:96) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.callPreActiveCallbacks(ActiveMQServerImpl.java:3117) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart1(ActiveMQServerImpl.java:3263) > - locked <0x000000009fb77060> (a > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) > at > > org.apache.activemq.artemis.core.server.impl.SharedStorePrimaryActivation.run(SharedStorePrimaryActivation.java:66) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:742) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:632) > - locked <0x000000009fb77060> (a > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) > at > > org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:66) > - locked <0x00000000809d94e8> (a > org.apache.activemq.artemis.integration.FileBroker) > at > org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:130) > at > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:222) > at > org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:168) > at > jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17.0.13 > /Native > Method) > at > jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17.0.13 > /NativeMethodAccessorImpl.java:77) > at > jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.13 > /DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(java.base@17.0.13 > /Method.java:569) > at > org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:152) > at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:64) > > The underlying javax.management.remote.rmi.RMIConnectorServer is > attempting to "bind" which appears to involve creating a Socket and reading > some data from it. > > I've tried reproducing this by specifying "-Djava.rmi.server.hostname=foo" > and: > > <connector connector-port="1099" connector-host="0.0.0.0" > rmi-registry-port="1100" /> > > However, there is no delay in broker start-up. > > Do you know of any way I can reproduce what you're seeing? > > > Justin > > [1] https://p.defau.lt/?WV75z6Vajriy_tmBpASMrg > > On Thu, Jan 23, 2025 at 3:50 AM Vilius Šumskas <vilius.sums...@rivile.lt> > wrote: > > > >> ...the broker hangs on startup for ~2 minutes and 10 seconds every > > time... > > > > > Can you get a thread dump during the delay and provide it here? The > > logging doesn't really indicate what's going on behind the scenes. I > > just did a quick proof-of-concept using a similar configuration to > > yours and I didn't see any delay. > > > > Sure. This is a thread dump during the delay: > > https://p.defau.lt/?WV75z6Vajriy_tmBpASMrg > > This is a thread dump just right after delay passes: > > https://p.defau.lt/?_yKMZJpdrGhfAzc2cy_t6A > > And this is a thread dump when broker starts completely: > > https://p.defau.lt/?x5MSCZE9kZknAEAKi1UORg > > > > If I'm reading it correctly thread "RMI RenewClean-[35.214.206.170:1100 > ]" > > #26 is waiting for something. As mentioned, the IP here is the > > external NAT IP of the host, i.e. it is not assigned to any interface on > the broker. > > > > -- > > Vilius > > > > -----Original Message----- > > From: Justin Bertram <jbert...@apache.org> > > Sent: Wednesday, January 22, 2025 7:16 PM > > To: users@activemq.apache.org > > Subject: Re: JMX exporter in Docker image > > > > > ...I had issues with configuration so decided to look into JMX > > > exporter > > as a workaround. > > > > To be clear, the "JMX exporter" [1] is a tool provided by Prometheus > > to read values from JMX MBeans and export them to a format which > > Prometheus can consume (i.e. scrape via HTTP). It has nothing to do > > with allowing remote JMX connectivity to a JVM which appears to be what > you really want. > > > > > Setting <connector connector-port="1099" rmi-registry-port="1099" /> > > produces "port already used" bind errors. > > > > It's possible to use the same values here when using the JVM's JMX > > connectivity so I expect it should be possible to do the same with our > > MBean server as well (although I'm not sure specifically how at this > > point). Feel free to open a ticket for this. > > > > > ...the broker hangs on startup for ~2 minutes and 10 seconds every > > time... > > > > Can you get a thread dump during the delay and provide it here? The > > logging doesn't really indicate what's going on behind the scenes. I > > just did a quick proof-of-concept using a similar configuration to > > yours and I didn't see any delay. > > > > > Some JMX software, for example VisualVM, cannot build its window for > > > JMX > > browsing unless vm* methods are added to role-access list. > > > > Can you provide more details about this? I just started up VisualVM > > and created a new (remote) JMX connection to a broker and it was able > > to browse MBeans no problem. > > > > > > Justin > > > > [1] https://github.com/prometheus/jmx_exporter > > > > On Tue, Jan 14, 2025 at 5:31 PM Vilius Šumskas > > <vilius.sums...@rivile.lt> > > wrote: > > > > > Yes, I've read documentation regarding JMX management, but I had > > > issues with configuration so decided to look into JMX exporter as a > > workaround. > > > > > > I also saw your suggestion regarding Prometheus plugin in another > > > thread, and will definitely try it, but I wanted to try JMX first as > > > there are much more monitoring and alerting dashboard templates > > > available. Most of the templates use JMX metric keys, so I thought > > > this would save me some time later. > > > > > > Anyway, my original issues with JMX are: > > > 1. Setting <connector connector-port="1099" rmi-registry-port="1099" > > > /> produces "port already used" bind errors. I can only set these > > > parameters to two different ports. It's not a big deal, but one > > > needs to know that both ports needs to be allowed in the firewall > > > then, and we have to keep in mind that RMI registry port traffic is > > > plaintext by > > default. > > > 2. If I set <connector connector-port="1099" connector-host="0.0.0.0" > > > rmi-registry-port="1100" /> and add > > > -Djava.rmi.server.hostname=<publicIPbeforeNAT> to artemis.profile, > > > the broker hangs on startup for ~2 minutes and 10 seconds every time > at: > > > 2025-01-14 22:49:38,308 INFO > > > [org.apache.activemq.artemis.core.server] > > > AMQ221006: Waiting to obtain primary lock > > > 2025-01-14 22:51:50,703 INFO > > > [org.apache.activemq.artemis.core.server] > > > AMQ221012: Using AIO Journal > > > > > > Looks like some kind of TCP timeout in play. Ports can match and I > > > don't experience this delay if I use standard JVM agent instead, i.e. > > > completely disable authorization and custom connector in > > > management.xml, as suggested in the documentation, and use: > > > -Dcom.sun.management.jmxremote.ssl=false > > > -Dcom.sun.management.jmxremote.authenticate=false > > > -Dcom.sun.management.jmxremote.port=1099 > > > -Dcom.sun.management.jmxremote.rmi.port=1099 > > > -Djava.rmi.server.hostname=<publicIPbeforeNAT> > > > -Dcom.sun.management.jmxremote.local.only=false > > > These first two issues are only reproducible with Artemis' custom > > > JMX connector. > > > > > > 3. Some JMX software, for example VisualVM, cannot build its window > > > for JMX browsing unless vm* methods are added to role-access list. I > > > didn't investigate further because of issue #2 it's too slow to > > > debug, but I guess it's missing access to some informational > > > actions, like > > maybe "vmInfo". > > > > > > Should I create tickets for all these issues? > > > > > > -- > > > Vilius > > > > > > -----Original Message----- > > > From: Justin Bertram <jbert...@apache.org> > > > Sent: Tuesday, January 14, 2025 7:22 PM > > > To: users@activemq.apache.org > > > Subject: Re: JMX exporter in Docker image > > > > > > You can enable remote JMX connectivity by modifying management.xml > > > as described in this documentation [1]. Of course you'd need to > > > expose whatever port you're using for remote JMX via Docker. > > > > > > The thing with port 9404 does appear to be a bug. These days we > > > recommend folks use the pluggable broker metrics [2] (for which > > > there is a Prometheus plugin - accessible via the web port 8161). > > > > > > > > > Justin > > > > > > [1] > > > > > > https://activemq.apache.org/components/artemis/documentation/latest/ > > > ma > > > nagement.html#remote-jmx-access > > > [2] > > > > > > https://activemq.apache.org/components/artemis/documentation/latest/ > > > me > > > trics.html#metrics > > > > > > On Tue, Jan 14, 2025 at 3:37 AM Vilius Šumskas > > > <vilius.sums...@rivile.lt> > > > wrote: > > > > > > > Hi, > > > > > > > > I‘m trying to configure JMX remoting in official Artemis Docker > image. > > > > I see that Docker image has port 9404 exposed which indicates JMX > > > > Prometheus exporter, but I cannot find any information how to > > > > activate it or even the exporter JAR in the image itself. Is this > > > > a bug left from > > > > > https://github.com/vromero/activemq-artemis-docker/blob/master/README. > > > > md#57-prometheus-metrics > > > > or I’m missing something? > > > > > > > > -- > > > > Vilius > > > > > > > > > > > > > > -------------------------------------------------------------------- > > > - To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > > > For additional commands, e-mail: users-h...@activemq.apache.org For > > > further information, visit: https://activemq.apache.org/contact > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > > For additional commands, e-mail: users-h...@activemq.apache.org For > > further information, visit: https://activemq.apache.org/contact > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > For additional commands, e-mail: users-h...@activemq.apache.org > For further information, visit: https://activemq.apache.org/contact > >
