Can you try with -Djava.rmi.server.hostname=some.random.ip instead of hostname?
I tried with "foo" hostname and there is no delay, but switched to 8.8.8.8 and
delay is back again. Actually specifying any IP not under host's interface
produces the same result.
--
Vilius
-----Original Message-----
From: Justin Bertram <jbert...@apache.org>
Sent: Thursday, January 23, 2025 5:58 PM
To: users@activemq.apache.org
Subject: Re: JMX exporter in Docker image
From what I can tell the delay is shown in the first thread dump [1]:
"main" #1 prio=5 os_prio=0 cpu=2709.38ms elapsed=33.95s
tid=0x00007f723002ae60 nid=0x33633 runnable [0x00007f7237121000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.Net.poll(java.base@17.0.13/Native Method)
at sun.nio.ch.NioSocketImpl.park(java.base@17.0.13
/NioSocketImpl.java:186)
at sun.nio.ch.NioSocketImpl.park(java.base@17.0.13
/NioSocketImpl.java:195)
at sun.nio.ch.NioSocketImpl.implRead(java.base@17.0.13
/NioSocketImpl.java:319)
at sun.nio.ch.NioSocketImpl.read(java.base@17.0.13
/NioSocketImpl.java:355)
at sun.nio.ch.NioSocketImpl$1.read(java.base@17.0.13
/NioSocketImpl.java:808)
at java.net.Socket$SocketInputStream.read(java.base@17.0.13
/Socket.java:966)
at java.io.BufferedInputStream.fill(java.base@17.0.13
/BufferedInputStream.java:244)
at java.io.BufferedInputStream.read(java.base@17.0.13
/BufferedInputStream.java:263)
- locked <0x000000009f631900> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readUnsignedByte(java.base@17.0.13
/DataInputStream.java:288)
at java.io.DataInputStream.readByte(java.base@17.0.13
/DataInputStream.java:268)
at sun.rmi.transport.StreamRemoteCall.executeCall(java.rmi@17.0.13
/StreamRemoteCall.java:241)
at sun.rmi.server.UnicastRef.invoke(java.rmi@17.0.13
/UnicastRef.java:381)
at sun.rmi.registry.RegistryImpl_Stub.bind(java.rmi@17.0.13
/RegistryImpl_Stub.java:73)
at
com.sun.jndi.rmi.registry.RegistryContext.bind(jdk.naming.rmi@17.0.13
/RegistryContext.java:157)
at
com.sun.jndi.toolkit.url.GenericURLContext.bind(java.naming@17.0.13
/GenericURLContext.java:243)
at javax.naming.InitialContext.bind(java.naming@17.0.13
/InitialContext.java:417)
at
javax.management.remote.rmi.RMIConnectorServer.bind(java.management.rmi@17.0.13
/RMIConnectorServer.java:693)
at
javax.management.remote.rmi.RMIConnectorServer.start(java.management.rmi@17.0.13
/RMIConnectorServer.java:476)
- locked <0x000000009f7facf8> (a
javax.management.remote.rmi.RMIConnectorServer)
at
org.apache.activemq.artemis.core.server.management.ConnectorServerFactory.init(ConnectorServerFactory.java:226)
at
org.apache.activemq.artemis.core.server.management.ManagementConnector.start(ManagementConnector.java:96)
at
org.apache.activemq.artemis.core.server.management.ManagementContext.start(ManagementContext.java:56)
- locked <0x00000000809db358> (a
org.apache.activemq.artemis.core.server.management.ManagementContext)
at
org.apache.activemq.artemis.cli.commands.Run$1.preActivate(Run.java:96)
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.callPreActiveCallbacks(ActiveMQServerImpl.java:3117)
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart1(ActiveMQServerImpl.java:3263)
- locked <0x000000009fb77060> (a
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
at
org.apache.activemq.artemis.core.server.impl.SharedStorePrimaryActivation.run(SharedStorePrimaryActivation.java:66)
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:742)
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:632)
- locked <0x000000009fb77060> (a
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
at
org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:66)
- locked <0x00000000809d94e8> (a
org.apache.activemq.artemis.integration.FileBroker)
at
org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:130)
at
org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:222)
at
org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:168)
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17.0.13/Native
Method)
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17.0.13
/NativeMethodAccessorImpl.java:77)
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.13
/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@17.0.13
/Method.java:569)
at
org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:152)
at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:64)
The underlying javax.management.remote.rmi.RMIConnectorServer is attempting to
"bind" which appears to involve creating a Socket and reading some data from it.
I've tried reproducing this by specifying "-Djava.rmi.server.hostname=foo"
and:
<connector connector-port="1099" connector-host="0.0.0.0"
rmi-registry-port="1100" />
However, there is no delay in broker start-up.
Do you know of any way I can reproduce what you're seeing?
Justin
[1] https://p.defau.lt/?WV75z6Vajriy_tmBpASMrg
On Thu, Jan 23, 2025 at 3:50 AM Vilius Šumskas <vilius.sums...@rivile.lt>
wrote:
> >> ...the broker hangs on startup for ~2 minutes and 10 seconds every
> time...
>
> > Can you get a thread dump during the delay and provide it here? The
> logging doesn't really indicate what's going on behind the scenes. I
> just did a quick proof-of-concept using a similar configuration to
> yours and I didn't see any delay.
>
> Sure. This is a thread dump during the delay:
> https://p.defau.lt/?WV75z6Vajriy_tmBpASMrg
> This is a thread dump just right after delay passes:
> https://p.defau.lt/?_yKMZJpdrGhfAzc2cy_t6A
> And this is a thread dump when broker starts completely:
> https://p.defau.lt/?x5MSCZE9kZknAEAKi1UORg
>
> If I'm reading it correctly thread "RMI RenewClean-[35.214.206.170:1100]"
> #26 is waiting for something. As mentioned, the IP here is the
> external NAT IP of the host, i.e. it is not assigned to any interface on the
> broker.
>
> --
> Vilius
>
> -----Original Message-----
> From: Justin Bertram <jbert...@apache.org>
> Sent: Wednesday, January 22, 2025 7:16 PM
> To: users@activemq.apache.org
> Subject: Re: JMX exporter in Docker image
>
> > ...I had issues with configuration so decided to look into JMX
> > exporter
> as a workaround.
>
> To be clear, the "JMX exporter" [1] is a tool provided by Prometheus
> to read values from JMX MBeans and export them to a format which
> Prometheus can consume (i.e. scrape via HTTP). It has nothing to do
> with allowing remote JMX connectivity to a JVM which appears to be what you
> really want.
>
> > Setting <connector connector-port="1099" rmi-registry-port="1099" />
> produces "port already used" bind errors.
>
> It's possible to use the same values here when using the JVM's JMX
> connectivity so I expect it should be possible to do the same with our
> MBean server as well (although I'm not sure specifically how at this
> point). Feel free to open a ticket for this.
>
> > ...the broker hangs on startup for ~2 minutes and 10 seconds every
> time...
>
> Can you get a thread dump during the delay and provide it here? The
> logging doesn't really indicate what's going on behind the scenes. I
> just did a quick proof-of-concept using a similar configuration to
> yours and I didn't see any delay.
>
> > Some JMX software, for example VisualVM, cannot build its window for
> > JMX
> browsing unless vm* methods are added to role-access list.
>
> Can you provide more details about this? I just started up VisualVM
> and created a new (remote) JMX connection to a broker and it was able
> to browse MBeans no problem.
>
>
> Justin
>
> [1] https://github.com/prometheus/jmx_exporter
>
> On Tue, Jan 14, 2025 at 5:31 PM Vilius Šumskas
> <vilius.sums...@rivile.lt>
> wrote:
>
> > Yes, I've read documentation regarding JMX management, but I had
> > issues with configuration so decided to look into JMX exporter as a
> workaround.
> >
> > I also saw your suggestion regarding Prometheus plugin in another
> > thread, and will definitely try it, but I wanted to try JMX first as
> > there are much more monitoring and alerting dashboard templates
> > available. Most of the templates use JMX metric keys, so I thought
> > this would save me some time later.
> >
> > Anyway, my original issues with JMX are:
> > 1. Setting <connector connector-port="1099" rmi-registry-port="1099"
> > /> produces "port already used" bind errors. I can only set these
> > parameters to two different ports. It's not a big deal, but one
> > needs to know that both ports needs to be allowed in the firewall
> > then, and we have to keep in mind that RMI registry port traffic is
> > plaintext by
> default.
> > 2. If I set <connector connector-port="1099" connector-host="0.0.0.0"
> > rmi-registry-port="1100" /> and add
> > -Djava.rmi.server.hostname=<publicIPbeforeNAT> to artemis.profile,
> > the broker hangs on startup for ~2 minutes and 10 seconds every time at:
> > 2025-01-14 22:49:38,308 INFO
> > [org.apache.activemq.artemis.core.server]
> > AMQ221006: Waiting to obtain primary lock
> > 2025-01-14 22:51:50,703 INFO
> > [org.apache.activemq.artemis.core.server]
> > AMQ221012: Using AIO Journal
> >
> > Looks like some kind of TCP timeout in play. Ports can match and I
> > don't experience this delay if I use standard JVM agent instead, i.e.
> > completely disable authorization and custom connector in
> > management.xml, as suggested in the documentation, and use:
> > -Dcom.sun.management.jmxremote.ssl=false
> > -Dcom.sun.management.jmxremote.authenticate=false
> > -Dcom.sun.management.jmxremote.port=1099
> > -Dcom.sun.management.jmxremote.rmi.port=1099
> > -Djava.rmi.server.hostname=<publicIPbeforeNAT>
> > -Dcom.sun.management.jmxremote.local.only=false
> > These first two issues are only reproducible with Artemis' custom
> > JMX connector.
> >
> > 3. Some JMX software, for example VisualVM, cannot build its window
> > for JMX browsing unless vm* methods are added to role-access list. I
> > didn't investigate further because of issue #2 it's too slow to
> > debug, but I guess it's missing access to some informational
> > actions, like
> maybe "vmInfo".
> >
> > Should I create tickets for all these issues?
> >
> > --
> > Vilius
> >
> > -----Original Message-----
> > From: Justin Bertram <jbert...@apache.org>
> > Sent: Tuesday, January 14, 2025 7:22 PM
> > To: users@activemq.apache.org
> > Subject: Re: JMX exporter in Docker image
> >
> > You can enable remote JMX connectivity by modifying management.xml
> > as described in this documentation [1]. Of course you'd need to
> > expose whatever port you're using for remote JMX via Docker.
> >
> > The thing with port 9404 does appear to be a bug. These days we
> > recommend folks use the pluggable broker metrics [2] (for which
> > there is a Prometheus plugin - accessible via the web port 8161).
> >
> >
> > Justin
> >
> > [1]
> >
> > https://activemq.apache.org/components/artemis/documentation/latest/
> > ma
> > nagement.html#remote-jmx-access
> > [2]
> >
> > https://activemq.apache.org/components/artemis/documentation/latest/
> > me
> > trics.html#metrics
> >
> > On Tue, Jan 14, 2025 at 3:37 AM Vilius Šumskas
> > <vilius.sums...@rivile.lt>
> > wrote:
> >
> > > Hi,
> > >
> > > I‘m trying to configure JMX remoting in official Artemis Docker image.
> > > I see that Docker image has port 9404 exposed which indicates JMX
> > > Prometheus exporter, but I cannot find any information how to
> > > activate it or even the exporter JAR in the image itself. Is this
> > > a bug left from
> > > https://github.com/vromero/activemq-artemis-docker/blob/master/README.
> > > md#57-prometheus-metrics
> > > or I’m missing something?
> > >
> > > --
> > > Vilius
> > >
> > >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org
> > For additional commands, e-mail: users-h...@activemq.apache.org For
> > further information, visit: https://activemq.apache.org/contact
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org
> For additional commands, e-mail: users-h...@activemq.apache.org For
> further information, visit: https://activemq.apache.org/contact
>
>
