Prometheus on Apache Tomcat multiple tomcat Instances

[Melvin Oosterveen]

Hi all,

I'm currently working on Apache Tomcat stacks within my company and I would 
like to integrate Prometheus into our deployments. We have decided to go for 
the Advanced Configuration - Multiple Tomcat Instances on UNIX systems 
described in the RUNNING.txt.

Does anyone have some experience with the setup of the JMX Prometheus exporter 
using multiple Tomcat instances? My goal is to have a separate Java process 
(Prometheus) started when I start an Apache Tomcat instance.

So far I have downloaded the JMX 
exporter<https://github.com/prometheus/jmx_exporter> and added it to the 
created Tomcat Instance with a minimal Prometheus configuration.

```bash

[root@melvin prometheus]# cat config.yml

hostPort: localhost:9999

rules:

- pattern: ".*"

```

I have created a service file which allows me to start the Apache Tomcat 
instances separately from each other. Inside this configuration I tried to 
start the Prometheus Java process, but so far I have failed and the Java 
process won't expose itself on the configured port.
```bash

[root@melvin prometheus]# cat /etc/systemd/system/tomcat-10@.service

[Unit]

Description=Tomcat - instance %i

After=syslog.target network.target


[Service]

Type=forking


User=tomcat

Group=tomcat


WorkingDirectory=/var/tomcat/%i


Environment="JAVA_HOME=/etc/alternatives/jre_openjdk"

Environment="JAVA_OPTS=-Djava.security.egd=file:///dev/urandom 
-Djavaagent:/var/tomcat/tomcat10/lib/jmx_prometheus_javaagent-0.17.2.jar=10080:/var/tomcat/tomcat10/prometheus/config.yml"

Environment="CATALINA_PID=/var/tomcat/%i/run/tomcat.pid"

Environment="CATALINA_BASE=/var/tomcat/%i/"

Environment="CATALINA_HOME=/opt/tomcat/10/"

Environment="CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC"


ExecStart=/opt/tomcat/10/bin/startup.sh

ExecStop=/opt/tomcat/10/bin/shutdown.sh


#RestartSec=10

#Restart=always


[Install]

WantedBy=multi-user.target

```

The instance then starts without errors and starts my sample.war application 
which I have downloaded from Apache.org. The initiated Java process is even 
shown.
```bash

tomcat    115910       1  0 Jan12 ?        00:02:24 
/etc/alternatives/jre_openjdk/bin/java 
-Djava.util.logging.config.file=/var/tomcat/tomcat10//conf/logging.properties 
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager 
-Djava.security.egd=file:///dev/urandom 
-Djavaagent:/var/tomcat/tomcat10/lib/jmx_prometheus_javaagent-0.17.2.jar=10080:/var/tomcat/tomcat10/prometheus/config.yml
 -Djdk.tls.ephemeralDHKeySize=2048 
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources 
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xms512M -Xmx1024M 
-server -XX:+UseParallelGC -Dignore.endorsed.dirs= -classpath 
/opt/tomcat/10//bin/bootstrap.jar:/var/tomcat/tomcat10//bin/tomcat-juli.jar 
-Dcatalina.base=/var/tomcat/tomcat10/ -Dcatalina.home=/opt/tomcat/10/ 
-Djava.io.tmpdir=/var/tomcat/tomcat10//temp 
org.apache.catalina.startup.Bootstrap start

```

I wonder if anyone could help me out with the right setup of Prometheus on 
Apache Tomcat multiple tomcat Instances.

Thanks in advance,

Melvin Oosterveen