Thanks. Created ARTEMIS-5260 and ARTEMIS-5261 regarding port and hostname
issues.
As for VisualVM JMX browsing issue, it's difficult to debug because changing
various permissions produces different and, I would say, strange results.
Sometimes VisualVM window just shows "Reloading information...", sometimes it
freezes with Java exception, sometimes it hangs for 1-2 minutes and then it
shows initial information about JVM allowing to browse Artemis' Mbeans.
This is on VisualVM 2.1.10 running on Windows 10 and Artemis 2.39.0 running on
Java 17. I'm also using "VisualVM-MBeans" plugin for VisualVM, if that makes a
difference to reproduce.
Anyway, I think I have found minimal required permissions for VisualVM to
connect and browse MBeans.
<match domain="com.sun.management">
<access method="help" roles="amq"/>
<access method="jfrCheck" roles="amq"/>
<access method="vmCommandLine" roles="amq"/>
</match>
in <role-access> should do the trick.
This is also looks more or less inline what is called here
https://github.com/oracle/visualvm/blob/7a9369351885aa07d5916fc530f9fcec183776ca/visualvm/jmx/src/org/graalvm/visualvm/jmx/impl/JmxSupport.java
and here
https://github.com/oracle/visualvm/blob/7a9369351885aa07d5916fc530f9fcec183776ca/visualvm/jfr/src/org/graalvm/visualvm/jfr/model/impl/JfrModelImpl.java
.
There is also an additional exception about VisualVM unable to call
gcClassHistogram, but since it doesn't block other VisualVM operations and I
didn't know if that's not too sensitive to allow by default, I didn't add it.
ARTEMIS-5262 is also created to track this issue.
--
Best Regards,
Vilius
-----Original Message-----
From: Justin Bertram <jbert...@apache.org>
Sent: Friday, January 24, 2025 8:08 PM
To: users@activemq.apache.org
Subject: Re: JMX exporter in Docker image
I am able to reproduce the delay using 8.8.8.8. I'm not sure how the JVM's
MBean server avoids this. Feel free to open a Jira to track this.
Justin
On Thu, Jan 23, 2025 at 11:34 AM Vilius Šumskas <vilius.sums...@rivile.lt>
wrote:
> Can you try with -Djava.rmi.server.hostname=some.random.ip instead of
> hostname?
>
> I tried with "foo" hostname and there is no delay, but switched to
> 8.8.8.8 and delay is back again. Actually specifying any IP not under
> host's interface produces the same result.
>
> --
> Vilius
>
> -----Original Message-----
> From: Justin Bertram <jbert...@apache.org>
> Sent: Thursday, January 23, 2025 5:58 PM
> To: users@activemq.apache.org
> Subject: Re: JMX exporter in Docker image
>
> From what I can tell the delay is shown in the first thread dump [1]:
>
> "main" #1 prio=5 os_prio=0 cpu=2709.38ms elapsed=33.95s
> tid=0x00007f723002ae60 nid=0x33633 runnable [0x00007f7237121000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.Net.poll(java.base@17.0.13/Native Method)
> at sun.nio.ch.NioSocketImpl.park(java.base@17.0.13
> /NioSocketImpl.java:186)
> at sun.nio.ch.NioSocketImpl.park(java.base@17.0.13
> /NioSocketImpl.java:195)
> at sun.nio.ch.NioSocketImpl.implRead(java.base@17.0.13
> /NioSocketImpl.java:319)
> at sun.nio.ch.NioSocketImpl.read(java.base@17.0.13
> /NioSocketImpl.java:355)
> at sun.nio.ch.NioSocketImpl$1.read(java.base@17.0.13
> /NioSocketImpl.java:808)
> at java.net.Socket$SocketInputStream.read(java.base@17.0.13
> /Socket.java:966)
> at java.io.BufferedInputStream.fill(java.base@17.0.13
> /BufferedInputStream.java:244)
> at java.io.BufferedInputStream.read(java.base@17.0.13
> /BufferedInputStream.java:263)
> - locked <0x000000009f631900> (a java.io.BufferedInputStream)
> at java.io.DataInputStream.readUnsignedByte(java.base@17.0.13
> /DataInputStream.java:288)
> at java.io.DataInputStream.readByte(java.base@17.0.13
> /DataInputStream.java:268)
> at
> sun.rmi.transport.StreamRemoteCall.executeCall(java.rmi@17.0.13
> /StreamRemoteCall.java:241)
> at sun.rmi.server.UnicastRef.invoke(java.rmi@17.0.13
> /UnicastRef.java:381)
> at sun.rmi.registry.RegistryImpl_Stub.bind(java.rmi@17.0.13
> /RegistryImpl_Stub.java:73)
> at
> com.sun.jndi.rmi.registry.RegistryContext.bind(jdk.naming.rmi@17.0.13
> /RegistryContext.java:157)
> at
> com.sun.jndi.toolkit.url.GenericURLContext.bind(java.naming@17.0.13
> /GenericURLContext.java:243)
> at javax.naming.InitialContext.bind(java.naming@17.0.13
> /InitialContext.java:417)
> at
>
> javax.management.remote.rmi.RMIConnectorServer.bind(java.management.rm
> i@17.0.13
> /RMIConnectorServer.java:693)
> at
>
> javax.management.remote.rmi.RMIConnectorServer.start(java.management.r
> mi@17.0.13
> /RMIConnectorServer.java:476)
> - locked <0x000000009f7facf8> (a
> javax.management.remote.rmi.RMIConnectorServer)
> at
>
> org.apache.activemq.artemis.core.server.management.ConnectorServerFactory.init(ConnectorServerFactory.java:226)
> at
>
> org.apache.activemq.artemis.core.server.management.ManagementConnector.start(ManagementConnector.java:96)
> at
>
> org.apache.activemq.artemis.core.server.management.ManagementContext.start(ManagementContext.java:56)
> - locked <0x00000000809db358> (a
> org.apache.activemq.artemis.core.server.management.ManagementContext)
> at
> org.apache.activemq.artemis.cli.commands.Run$1.preActivate(Run.java:96)
> at
>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.callPreActiveCallbacks(ActiveMQServerImpl.java:3117)
> at
>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart1(ActiveMQServerImpl.java:3263)
> - locked <0x000000009fb77060> (a
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
> at
>
> org.apache.activemq.artemis.core.server.impl.SharedStorePrimaryActivation.run(SharedStorePrimaryActivation.java:66)
> at
>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:742)
> at
>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:632)
> - locked <0x000000009fb77060> (a
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
> at
>
> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:66)
> - locked <0x00000000809d94e8> (a
> org.apache.activemq.artemis.integration.FileBroker)
> at
> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:130)
> at
> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:222)
> at
> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:168)
> at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17.0.1
> 3
> /Native
> Method)
> at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17.0.13
> /NativeMethodAccessorImpl.java:77)
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.
> 0.13
> /DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(java.base@17.0.13
> /Method.java:569)
> at
> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:152)
> at
> org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:64)
>
> The underlying javax.management.remote.rmi.RMIConnectorServer is
> attempting to "bind" which appears to involve creating a Socket and
> reading some data from it.
>
> I've tried reproducing this by specifying "-Djava.rmi.server.hostname=foo"
> and:
>
> <connector connector-port="1099" connector-host="0.0.0.0"
> rmi-registry-port="1100" />
>
> However, there is no delay in broker start-up.
>
> Do you know of any way I can reproduce what you're seeing?
>
>
> Justin
>
> [1] https://p.defau.lt/?WV75z6Vajriy_tmBpASMrg
>
> On Thu, Jan 23, 2025 at 3:50 AM Vilius Šumskas
> <vilius.sums...@rivile.lt>
> wrote:
>
> > >> ...the broker hangs on startup for ~2 minutes and 10 seconds
> > >> every
> > time...
> >
> > > Can you get a thread dump during the delay and provide it here?
> > > The
> > logging doesn't really indicate what's going on behind the scenes. I
> > just did a quick proof-of-concept using a similar configuration to
> > yours and I didn't see any delay.
> >
> > Sure. This is a thread dump during the delay:
> > https://p.defau.lt/?WV75z6Vajriy_tmBpASMrg
> > This is a thread dump just right after delay passes:
> > https://p.defau.lt/?_yKMZJpdrGhfAzc2cy_t6A
> > And this is a thread dump when broker starts completely:
> > https://p.defau.lt/?x5MSCZE9kZknAEAKi1UORg
> >
> > If I'm reading it correctly thread "RMI
> > RenewClean-[35.214.206.170:1100
> ]"
> > #26 is waiting for something. As mentioned, the IP here is the
> > external NAT IP of the host, i.e. it is not assigned to any
> > interface on
> the broker.
> >
> > --
> > Vilius
> >
> > -----Original Message-----
> > From: Justin Bertram <jbert...@apache.org>
> > Sent: Wednesday, January 22, 2025 7:16 PM
> > To: users@activemq.apache.org
> > Subject: Re: JMX exporter in Docker image
> >
> > > ...I had issues with configuration so decided to look into JMX
> > > exporter
> > as a workaround.
> >
> > To be clear, the "JMX exporter" [1] is a tool provided by Prometheus
> > to read values from JMX MBeans and export them to a format which
> > Prometheus can consume (i.e. scrape via HTTP). It has nothing to do
> > with allowing remote JMX connectivity to a JVM which appears to be
> > what
> you really want.
> >
> > > Setting <connector connector-port="1099" rmi-registry-port="1099"
> > > />
> > produces "port already used" bind errors.
> >
> > It's possible to use the same values here when using the JVM's JMX
> > connectivity so I expect it should be possible to do the same with
> > our MBean server as well (although I'm not sure specifically how at
> > this point). Feel free to open a ticket for this.
> >
> > > ...the broker hangs on startup for ~2 minutes and 10 seconds every
> > time...
> >
> > Can you get a thread dump during the delay and provide it here? The
> > logging doesn't really indicate what's going on behind the scenes. I
> > just did a quick proof-of-concept using a similar configuration to
> > yours and I didn't see any delay.
> >
> > > Some JMX software, for example VisualVM, cannot build its window
> > > for JMX
> > browsing unless vm* methods are added to role-access list.
> >
> > Can you provide more details about this? I just started up VisualVM
> > and created a new (remote) JMX connection to a broker and it was
> > able to browse MBeans no problem.
> >
> >
> > Justin
> >
> > [1] https://github.com/prometheus/jmx_exporter
> >
> > On Tue, Jan 14, 2025 at 5:31 PM Vilius Šumskas
> > <vilius.sums...@rivile.lt>
> > wrote:
> >
> > > Yes, I've read documentation regarding JMX management, but I had
> > > issues with configuration so decided to look into JMX exporter as
> > > a
> > workaround.
> > >
> > > I also saw your suggestion regarding Prometheus plugin in another
> > > thread, and will definitely try it, but I wanted to try JMX first
> > > as there are much more monitoring and alerting dashboard templates
> > > available. Most of the templates use JMX metric keys, so I thought
> > > this would save me some time later.
> > >
> > > Anyway, my original issues with JMX are:
> > > 1. Setting <connector connector-port="1099" rmi-registry-port="1099"
> > > /> produces "port already used" bind errors. I can only set these
> > > parameters to two different ports. It's not a big deal, but one
> > > needs to know that both ports needs to be allowed in the firewall
> > > then, and we have to keep in mind that RMI registry port traffic
> > > is plaintext by
> > default.
> > > 2. If I set <connector connector-port="1099" connector-host="0.0.0.0"
> > > rmi-registry-port="1100" /> and add
> > > -Djava.rmi.server.hostname=<publicIPbeforeNAT> to artemis.profile,
> > > the broker hangs on startup for ~2 minutes and 10 seconds every
> > > time
> at:
> > > 2025-01-14 22:49:38,308 INFO
> > > [org.apache.activemq.artemis.core.server]
> > > AMQ221006: Waiting to obtain primary lock
> > > 2025-01-14 22:51:50,703 INFO
> > > [org.apache.activemq.artemis.core.server]
> > > AMQ221012: Using AIO Journal
> > >
> > > Looks like some kind of TCP timeout in play. Ports can match and I
> > > don't experience this delay if I use standard JVM agent instead, i.e.
> > > completely disable authorization and custom connector in
> > > management.xml, as suggested in the documentation, and use:
> > > -Dcom.sun.management.jmxremote.ssl=false
> > > -Dcom.sun.management.jmxremote.authenticate=false
> > > -Dcom.sun.management.jmxremote.port=1099
> > > -Dcom.sun.management.jmxremote.rmi.port=1099
> > > -Djava.rmi.server.hostname=<publicIPbeforeNAT>
> > > -Dcom.sun.management.jmxremote.local.only=false
> > > These first two issues are only reproducible with Artemis' custom
> > > JMX connector.
> > >
> > > 3. Some JMX software, for example VisualVM, cannot build its
> > > window for JMX browsing unless vm* methods are added to
> > > role-access list. I didn't investigate further because of issue #2
> > > it's too slow to debug, but I guess it's missing access to some
> > > informational actions, like
> > maybe "vmInfo".
> > >
> > > Should I create tickets for all these issues?
> > >
> > > --
> > > Vilius
> > >
> > > -----Original Message-----
> > > From: Justin Bertram <jbert...@apache.org>
> > > Sent: Tuesday, January 14, 2025 7:22 PM
> > > To: users@activemq.apache.org
> > > Subject: Re: JMX exporter in Docker image
> > >
> > > You can enable remote JMX connectivity by modifying management.xml
> > > as described in this documentation [1]. Of course you'd need to
> > > expose whatever port you're using for remote JMX via Docker.
> > >
> > > The thing with port 9404 does appear to be a bug. These days we
> > > recommend folks use the pluggable broker metrics [2] (for which
> > > there is a Prometheus plugin - accessible via the web port 8161).
> > >
> > >
> > > Justin
> > >
> > > [1]
> > >
> > > https://activemq.apache.org/components/artemis/documentation/lates
> > > t/
> > > ma
> > > nagement.html#remote-jmx-access
> > > [2]
> > >
> > > https://activemq.apache.org/components/artemis/documentation/lates
> > > t/
> > > me
> > > trics.html#metrics
> > >
> > > On Tue, Jan 14, 2025 at 3:37 AM Vilius Šumskas
> > > <vilius.sums...@rivile.lt>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I‘m trying to configure JMX remoting in official Artemis Docker
> image.
> > > > I see that Docker image has port 9404 exposed which indicates
> > > > JMX Prometheus exporter, but I cannot find any information how
> > > > to activate it or even the exporter JAR in the image itself. Is
> > > > this a bug left from
> > > >
> https://github.com/vromero/activemq-artemis-docker/blob/master/README.
> > > > md#57-prometheus-metrics
> > > > or I’m missing something?
> > > >
> > > > --
> > > > Vilius
> > > >
> > > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org
> > > For additional commands, e-mail: users-h...@activemq.apache.org
> > > For further information, visit:
> > > https://activemq.apache.org/contact
> > >
> > >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org
> > For additional commands, e-mail: users-h...@activemq.apache.org For
> > further information, visit: https://activemq.apache.org/contact
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org
> For additional commands, e-mail: users-h...@activemq.apache.org For
> further information, visit: https://activemq.apache.org/contact
>
>
