I have activemq configured with a durable subscriber to a topic backed by oracle. I left a publisher running at a rate of 100 msgs/sec; each msg is 1K. The msgs expire in 30 secs. The durable subscriber disconnected so, as expected, messages queued up in the db. After about 25 minutes I got an OutOfMemory error in my log; eventually the broker became unresponsive.
According to jstat permgen space usage is at 96% (!). There are no permgen messages in the logs though. My questions: 1. I had lowered the max heap space to 512M as a test -- the jstat output makes me thing this is ok, but maybe I need to boost permgen space? 2. How reliable/useful is this Health/CurrentStatus mbean attribute? (see below) 3. I know it is a longshot given the details I can provide, but any clues or tips on what went wrong for me? I'll see if I can reproduce the problem. == More details There are 320k messages loitering in the database; I suspect that expiration is not working properly. Any special tricks to enabling expiration? There are no ERROR messages in the logs, but there are WARN level msgs. activemq.log:2014-02-06 22:58:41,527 | WARN | JDBC Failure: Protocol violation: [ 0, ] | org.apache.activemq.store.jdbc.JDBCPersistenceAdapter | ActiveMQ Broker[localhost] Scheduler ... activemq.log:2014-02-06 22:58:12,295 | WARN | Failed to browse Topic: fooTopic | org.apache.activemq.broker.region.Topic | ActiveMQ Broker[localhost] Scheduler There are some messages in the log but they do not have the log4j format so I'm not sure the application generated them: - java.lang.OutOfMemoryError: GC overhead limit exceeded - java.lang.OutOfMemoryError: Java heap space "jstat -gc 1000" shows a static situation and (to me) a healthy looking heap, but permgen space usage is at 96% (!). There are no permgen messages in the logs though. S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT 33536.0 32768.0 0.0 0.0 108544.0 16046.8 348160.0 286152.0 24576.0 23536.2 1943 71.415 7421 8187.680 8259.095 I was expecting to use the JMX Health MBean attribute "CurrentStatus" as a healthcheck datapoint for our monitoring system. Today though I managed to inadvertently induce an out of memory condition in activemq that seems to have also led to a JDBC failure. Connection attempts to the broker result in a "javax.jms.JMSException: Wire format negotiation timeout: peer did not send his wire format". Persistent messages are lingering in the jdbc datastore even though the messages are all expired. The RESTful endpoint is unresponsive. I used jmxterm to gain access to the activemq mbeans. Checking the "Health" mbean I was expecting to see something like a "Critical" status in the CurrentStatus attribute, but instead it says "Good". How reliable/useful is this Health/CurrentStatus mbean attribute? Here is my jmxterm output: $>bean org.apache.activemq:brokerName=localhost,service=Health,type=Broker #bean is set to org.apache.activemq:brokerName=localhost,service=Health,type=Broker $>info #mbean = org.apache.activemq:brokerName=localhost,service=Health,type=Broker #class name = org.apache.activemq.broker.jmx.HealthView # attributes %0 - CurrentStatus (java.lang.String, r) # operations %0 - javax.management.openmbean.TabularData health() %1 - java.util.List healthList() #there's no notifications $>get CurrentStatus #mbean = org.apache.activemq:brokerName=localhost,service=Health,type=Broker: CurrentStatus = Good; -- View this message in context: http://activemq.2283324.n4.nabble.com/Out-of-memory-Permgen-JMX-attribute-Health-CurrentStatus-tp4677636.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
