Hello Christopher.,
Thanks for the Response.
Firstly thanks for adding a point me in asking me to check, if the annotations
are reflecting in the Java process, which opened me a door to add the concerned
annotations in correct place., by adding in java_tool_options in stead of
Java_opts.
yeah they are reflecting and creating a Java Process.
but I am facing a problem here., while i am checking JSP, Thats : the port i
am using here to enable JMX is been opening a process with the mentioned port
and at the same time shows port is being used.:
root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# jps
Picked up JAVA_TOOL_OPTIONS: -Xms2048M -Xmx10240M -XX:+UseStringDeduplication
-XX:+UseContainerSupport -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.port=49151
-Djava.rmi.server.hostname=tomcat.default.svc.cluster.local
-javaagent:/datadog-lib/dd-java-agent.jar
-XX:OnError=/datadog-lib/continuousprofiler/tmp/dd_crash_uploader.sh
-XX:ErrorFile=/datadog-lib/continuousprofiler/tmp/hs_err_pid_%p.log
Error: Exception thrown by the agent : java.rmi.server.ExportException: Port
already in use: 49151; nested exception is:
java.net.BindException: Address already in use (Bind failed)
jdk.internal.agent.AgentConfigurationError: java.rmi.server.ExportException:
Port already in use: 49151; nested exception is:
java.net.BindException: Address already in use (Bind failed)
at
jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:491)
at
jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:447)
at
jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:599)
Caused by: java.rmi.server.ExportException: Port already in use: 49151; nested
exception is:
java.net.BindException: Address already in use (Bind failed)
at
java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:335)
at
java.rmi/sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:243)
at
java.rmi/sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:412)
at java.rmi/sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
at
java.rmi/sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:234)
at java.rmi/sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:220)
at java.rmi/sun.rmi.registry.RegistryImpl.<init>(RegistryImpl.java:180)
at
jdk.management.agent/sun.management.jmxremote.SingleEntryRegistry.<init>(SingleEntryRegistry.java:49)
at
jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:836)
at
jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:479)
... 2 more
Caused by: java.net.BindException: Address already in use (Bind failed)
at java.base/java.net.PlainSocketImpl.socketBind(Native Method)
at
java.base/java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:452)
at java.base/java.net.ServerSocket.bind(ServerSocket.java:395)
at java.base/java.net.ServerSocket.<init>(ServerSocket.java:257)
at java.base/java.net.ServerSocket.<init>(ServerSocket.java:149)
at
java.rmi/sun.rmi.transport.tcp.TCPDirectSocketFactory.createServerSocket(TCPDirectSocketFactory.java:45)
at
java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:670)
at
java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:324)
... 11 more
------------------------------------------------------------------------
Java process is been referred as Default JAva process:
root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# ps -aux | grep
java
root 1 5.3 1.9 20332668 2525004 ? Ssl 07:42 3:41 java -jar
/app/workflow-service.jar
root 928 0.0 0.0 6328 684 pts/0 S+ 08:50 0:00 grep
--color=auto java
and port usage is as :
root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# netstat -tulnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
tcp6 0 0 :::10109 :::* LISTEN
1/java
tcp6 0 0 :::9109 :::* LISTEN
1/java
tcp6 0 0 :::49151 :::* LISTEN
1/java
tcp6 0 0 :::42965 :::* LISTEN
1/java
I have used an extreme port range, to make sure that the concerned port should
not be in listening list on the application end.
Please let me know if you have any suggestions me to refer from my end!
Thanks
Thanks & Regards,
--
SAI VAMSI .B
Senior DevOps Engineer
________________________________
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Thursday, November 30, 2023 04:07
To: users@tomcat.apache.org <users@tomcat.apache.org>
Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Sai Vamsi,
On 11/29/23 03:50, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
> I am trying to add Tomcat-JMX Integration to the Datadog Agent, in order to
> achieve Remote Monitoring .
> I am following the docs
> https://urldefense.com/v3/__https://docs.datadoghq.com/containers/guide/autodiscovery-with-jmx/?tab=helm*autodiscovery-annotations__;Iw!!GFN0sa3rsbfR8OLyAw!ZWe640rbBZ7N396lgR23yoo6ALro2xOf6L5lX7cVrpEf19qb0yEfJkd-8jsUiTCABJ46H974kV-LzaS-pYo5EtQDEVC4MYm-FCIfzPlzzlXVqQ$
> [https://urldefense.com/v3/__https://datadog-docs.imgix.net/images/dd-docs-meta-image.png__;!!GFN0sa3rsbfR8OLyAw!ZWe640rbBZ7N396lgR23yoo6ALro2xOf6L5lX7cVrpEf19qb0yEfJkd-8jsUiTCABJ46H974kV-LzaS-pYo5EtQDEVC4MYm-FCIfzPnR3EJUUg$
>
> ]<https://urldefense.com/v3/__https://docs.datadoghq.com/containers/guide/autodiscovery-with-jmx/?tab=helm*autodiscovery-annotations__;Iw!!GFN0sa3rsbfR8OLyAw!ZWe640rbBZ7N396lgR23yoo6ALro2xOf6L5lX7cVrpEf19qb0yEfJkd-8jsUiTCABJ46H974kV-LzaS-pYo5EtQDEVC4MYm-FCIfzPlzzlXVqQ$
> >
> Autodiscovery with
> JMX<https://urldefense.com/v3/__https://docs.datadoghq.com/containers/guide/autodiscovery-with-jmx/?tab=helm*autodiscovery-annotations__;Iw!!GFN0sa3rsbfR8OLyAw!ZWe640rbBZ7N396lgR23yoo6ALro2xOf6L5lX7cVrpEf19qb0yEfJkd-8jsUiTCABJ46H974kV-LzaS-pYo5EtQDEVC4MYm-FCIfzPlzzlXVqQ$
> >
> Datadog, the leading service for cloud-scale monitoring.
> docs.datadoghq.com
>
>
> This is the Environment I am using in my case.
> I am trying to integrate JMX in one my Deployment, in AKS environment and I
> am using Helm to upgrade my environment.
> as suggested in the documentation., I am adding the annotations in my
> deployment file in helm structure.
Okay, I know absolutely nothing about Azure, Kubernetes, or Helm. But it
looks like maybe this is more basic than that...
> My agent is JMX enabled, and i can see in deployment logs.
> I have configured a service for this deployment and its reflecting.
>
>
> Issue I am facing :
> I am trying to add my annotations, in the deployment file, after
> deployment, it should be able make the remote connection , but its showing
> these logs in the agent.
I will be of no help in figuring out what "annotations" you need in your
"deployment file" because - again - I know nothing about Azure, Kube,
and Helm.
> while i check the Java process., even its not reflecting this JMX process ,
> it seems JMX process is not at all created, in my case.,
>
> error i am facing after deployment"
> ========
> JMXFetch
> ========
>
> Information
> ==================
> runtime_version : 11.0.18
> version : 0.47.8
> Initialized checks
> ==================
> tomcat
> instance_name : tomcat-10.75.12.143-1099
> message : Unable to instantiate or initialize instance
> 10.75.12.143:1099. Is the target JMX Server or JVM running? Failed to
> retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root
> exception is java.rmi.ConnectException: Connection refused to host:
> 10.75.12.143; nested exception is:
> java.net.ConnectException: Connection refused (Connection refused)]
Well THAT seems straightforward.
First: is that the right IP address?
Second: on that server (10.75.12.143), is port 1099 bound-to by the
Tomcat process? Can you, for example, nc to that TCP/IP port? Have you
checked the Tomcat logs to see if there any errors during startup... for
example, any JMX-related errors?
Third: on that IP, is port 1099 reachable from the Helm (or whatever)
server that is trying to contact it? Try using "nc 10.75.12.143 1099"
and see if you can connect from Helm (or whatever).
> - name: JAVA_OPTS
> value: >-
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.local.only=false
> -Dcom.sun.management.jmxremote.port=1099
> -Dcom.sun.management.jmxremote.rmi.port=1099
> -Djava.rmi.server.hostname=$(POD_IP)
> -Djava.net.preferIPv4Stack=true
> -Dcom.sun.management.jmxremote.host=$(POD_IP)
Can you login to 10.75.12.143 and check to see if the JVM launch command
has (a) all of the above -D arguments in it and (b) if the host/hostname
have the expected values in them?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
