When LDAP auth is enabled and this problem is occurring, could you please use VisualVM or another performance analyzer to capture a CPU sampling snapshot over a minute or so? That should allow us to understand what's so slow about the LDAP code path.
Thanks, Tim On Tue, Jan 25, 2022, 8:44 AM Guillaume Cripiau < guillaume.crip...@murprogroup.com> wrote: > Hello, > > Apache LDAP is used to give access to topic. When Apache LDAP is disable, > the device connect normally. When Apache LDAP is enable, the latency can > exceed one second and the devices have problems to connect because of their > timeout. We don't have this latency with another message broker. > > Here are CPU and memory load in attachment. > > We sometimes have centreon alert. See connection JMX captures in > attachment : > - CTA_2022-01-19_06-31.png -> 4 cores at 100%. CPU for activemq is at 389% > - CTA_2022-01-19_06-33.png -> after the reboot > > ActiveMQ config : > > *VM Summary* mercredi 19 janvier 2022 8 h 36 min 48 s CET > ------------------------------ > Connection name: > admin@service:jmx:rmi:///jndi/rmi://51.255.198.17:31616/jmxrmi > Virtual Machine: OpenJDK 64-Bit Server VM version 25.312-b07 > Vendor: Red Hat, Inc. > Name: 11...@vps391568.ovh.net > Uptime: 32 days 19 hours 56 minutes > Process CPU time: 1 hour 54 minutes > JIT compiler: HotSpot 64-Bit Tiered Compilers > Total compile time: 1 minute > ------------------------------ > Live threads: 59 > Peak: 70 > Daemon threads: 37 > Total threads started: 7.070 > Current classes loaded: 8.584 > Total classes loaded: 8.588 > Total classes unloaded: 4 > > > ------------------------------ > Current heap size: 565.627 kbytes > Maximum heap size: 1.864.192 kbytes > Committed memory: 953.856 kbytes > Pending finalization: 0 objects > Garbage collector: Name = 'PS MarkSweep', Collections = 2, Total time > spent = 0,180 seconds > Garbage collector: Name = 'PS Scavenge', Collections = 183, Total time > spent = 47,516 seconds > ------------------------------ > Operating System: Linux 3.10.0-957.21.3.el7.x86_64 > Architecture: amd64 > Number of processors: 2 > Committed virtual memory: 4.782.976 kbytes > Total physical memory: 7.814.608 kbytes > Free physical memory: 3.406.084 kbytes > Total swap space: 0 kbytes > Free swap space: 0 kbytes > ------------------------------ > VM arguments: -Xms1024M -Xmx2G > -Djava.util.logging.config.file=logging.properties > -Djava.security.auth.login.config=/opt/apache-activemq-5.16.3//conf/login.config > -Dorg.apache.activemq.audit=true -Djava.rmi.server.hostname=51.255.198.17 > -Dcom.sun.management.jmxremote.port=31616 > -Dcom.sun.management.jmxremote.password.file=/opt/apache-activemq-5.16.3//conf/jmx.password > -Dcom.sun.management.jmxremote.access.file=/opt/apache-activemq-5.16.3//conf/jmx.access > -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote > -Djava.awt.headless=true -Djava.io.tmpdir=/opt/apache-activemq-5.16.3//tmp > -Dactivemq.classpath=/opt/apache-activemq-5.16.3// > conf:/opt/apache-activemq-5.16.3//../lib/: > -Dactivemq.home=/opt/apache-activemq-5.16.3/ > -Dactivemq.base=/opt/apache-activemq-5.16.3/ > -Dactivemq.conf=/opt/apache-activemq-5.16.3//conf > -Dactivemq.data=/opt/apache-activemq-5.16.3//data > Class path: /opt/apache-activemq-5.16.3//bin/activemq.jar > Library path: > /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > Boot class path: > /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/resources.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/rt.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/jsse.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/jce.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/charsets.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/jfr.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/classes > > Regards > > Guillaume Cripiau > > Le 15-11-21 à 14:10, Tim Bain a écrit : > > What is the CPU load like on the broker host? Is that process spiking the > CPU? If so, can you use JVisualVM or similar to do CPU sampling of the > broker process and find out where it's spending its time? > > My suspicion is that the broker is so busy that it's not responding to some > connections fast enough to let them succeed. > > Tim > > On Wed, Nov 10, 2021, 2:17 AM Guillaume Cripiau > <guillaume.crip...@murprogroup.com> wrote: > > > Hello, > > The only other logs (mqtt messages on the server) show this : > 10:30:45 > remove/ID:airprotec-cta-41267-1635489094200-3:8407/03200646/Connection > 10:30:46 > add/ID:airprotec-cta-41267-1635489094200-3:8410/03200646/Connection > 10:30:46 > remove/ID:airprotec-cta-41267-1635489094200-3:8410/03200646/Connection > > Yes, the client restarts after 2 hours without connection and the same > event happens but not always. > > The clients try to reconnect, the broker has just restarted. > > We cannot reproduce the problem on our dev server with minimal setup. > > We actually analyse the TCP connexion : the clients use the source port > 1883 and the timeout between a CONNECT command and CONNACK is 1 second. > Can this settings be the cause of our problems? > > Regards > > Guillaume Cripiau > > Le 02-11-21 à 12:45, Tim Bain a écrit : > > These are broker logs, or client logs? Whichever it is, what's in the > > other > > process's logs at the same time? > > You said that reconnecting isn't possible for hours after this happens. > > Do > > you see the same messages in the logs for that whole time? > > At the time this happens, what are the client(s) and the broker doing? > > Can you reproduce the problem reliably in a minimal setup (just one or > > two > > clients, minimal time and message load)? > > Tim > > On Tue, Nov 2, 2021, 5:19 AM Guillaume Cripiau > <guillaume.crip...@murprogroup.com> wrote: > > > Hello, > > We need support. Many devices disconnect for no apparent reason and > cannot reconnect after several days. Could you help us solve this > problem? Here is an example in the logs : > > > 2021-11-02 10:30:44,413 | WARN | Transport Connection to: > tcp://62.83.10.35:1024 failed | > org.apache.activemq.broker.TransportConnection.Transport | Async > Exception Handler | localhost | | > java.io.IOException: Connection reset > at > > > > org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:46)[activemq-client-5.16.3.jar:5.16.3] > > at > > > > org.apache.activemq.transport.mqtt.MQTTTransportFilter.oneway(MQTTTransportFilter.java:83)[activemq-mqtt-5.16.3.jar:5.16.3] > > at > > > > org.apache.activemq.transport.mqtt.MQTTInactivityMonitor.doOnewaySend(MQTTInactivityMonitor.java:185)[activemq-mqtt-5.16.3.jar:5.16.3] > > at > > > > org.apache.activemq.transport.mqtt.MQTTInactivityMonitor.oneway(MQTTInactivityMonitor.java:174)[activemq-mqtt-5.16.3.jar:5.16.3] > > at > > > > org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)[activemq-client-5.16.3.jar:5.16.3] > > at > > > > org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1485)[activemq-broker-5.16.3.jar:5.16.3] > > at > > > > org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:982)[activemq-broker-5.16.3.jar:5.16.3] > > at > > > > org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:1033)[activemq-broker-5.16.3.jar:5.16.3] > > at > > > > org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)[activemq-client-5.16.3.jar:5.16.3] > > at > > > > org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)[activemq-client-5.16.3.jar:5.16.3] > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > Source)[:1.8.0_211] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source)[:1.8.0_211] > at java.lang.Thread.run(Unknown Source)[:1.8.0_211] > 2021-11-02 10:30:46,359 | WARN | Transport Connection to: > tcp://62.83.10.35:35598 failed | > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ > Transport: tcp:///62.83.10.35:35598@1883 | localhost | | > org.apache.activemq.transport.mqtt.MQTTProtocolException: Already > connected. > at > > > > org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTConnect(MQTTProtocolConverter.java:239)[activemq-mqtt-5.16.3.jar:5.16.3] > > at > > > > org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTCommand(MQTTProtocolConverter.java:204)[activemq-mqtt-5.16.3.jar:5.16.3] > > at > > > > org.apache.activemq.transport.mqtt.MQTTTransportFilter.onCommand(MQTTTransportFilter.java:94)[activemq-mqtt-5.16.3.jar:5.16.3] > > at > > > > org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.16.3.jar:5.16.3] > > at > > > > org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.16.3.jar:5.16.3] > > at > > > > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.16.3.jar:5.16.3] > > at java.lang.Thread.run(Unknown Source)[:1.8.0_211] > 2021-11-02 10:31:19,226 | WARN | Transport Connection to: > tcp://62.83.10.35:3000 failed | > org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ > Transport: tcp:///62.83.10.35:3000@1883 | localhost | | > org.apache.activemq.transport.mqtt.MQTTProtocolException: Not > > connected. > > at > > > > org.apache.activemq.transport.mqtt.MQTTProtocolConverter.checkConnected(MQTTProtocolConverter.java:736)[activemq-mqtt-5.16.3.jar:5.16.3] > > at > > > > org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onSubscribe(MQTTProtocolConverter.java:377)[activemq-mqtt-5.16.3.jar:5.16.3] > > at > > > > org.apache.activemq.transport.mqtt.MQTTProtocolConverter.onMQTTCommand(MQTTProtocolConverter.java:212)[activemq-mqtt-5.16.3.jar:5.16.3] > > at > > > > org.apache.activemq.transport.mqtt.MQTTTransportFilter.onCommand(MQTTTransportFilter.java:94)[activemq-mqtt-5.16.3.jar:5.16.3] > > at > > > > org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.16.3.jar:5.16.3] > > at > > > > org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)[activemq-client-5.16.3.jar:5.16.3] > > at > > > > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.16.3.jar:5.16.3] > > at java.lang.Thread.run(Unknown Source)[:1.8.0_211] > > Regards > > Guillaume Cripiau > > >
