A thread dump of the broker jvm, when the clients are stalled, would help diagnose what is going on. btw, what activemq version are you using?
2010/1/15 David Delbecq <de...@oma.be> > Le 15/01/10 10:24, Dejan Bosanac a écrit : > > Hi David, > > > > you're probably blocked by "producer flow control". Take a look here > > for more info: http://activemq.apache.org/producer-flow-control.html > > > > Cheers > > I already took a look at it, read it three time to be sure i understand. > It's quite technical but beyond that, i don't think this apply for me. > My queues are all empty!!! (they stalled when there were 50 messages > pending but where finally emptied), connections are not shared by > several queues and am not sure how to implement the cursor part of > documentation. There is no "slow consumer" that could influence it at > that right moment (customers are running an waiting for new messages as > you can see in capture). Moreover my messages are of about 1k payload > (about 10 small text entries in a MapMessage), and my activeMQ config is > like this > <policyEntry queue=">" memoryLimit="500mb"/> > <policyEntry topic=">" memoryLimit="500mb"> > .... > <systemUsage> > <systemUsage> > <memoryUsage> > <memoryUsage limit="20 mb"/> > </memoryUsage> > <storeUsage> > <storeUsage limit="1 gb" name="foo"/> > </storeUsage> > <tempUsage> > <tempUsage limit="100 mb"/> > </tempUsage> > </systemUsage> > </systemUsage> > > > There is no way a bunch of 10 pending messages could fill a 1gb store, > isn't it? What config part of activemq do i need to change to ensure > that it can store at least about 10000 persistent messages (that makes > only 10Mb of datas!!) before failing? Currently it fails and is never > recovering without a restart of broker when there are only 50 small > pending messages in the queue. As a workaround i have set > "sendFailIfNoSpace" to true, so my client will fail instead of stalling, > i'll see when this happen more quickly like that.... > > > Here is broker part of my config file, can someone help me fix this? > Thank you: > > > <broker xmlns="http://activemq.apache.org/schema/core"; > brokerName="localhost" dataDirectory="${activemq.base}/data" > persistent="true"> > <plugins> > <simpleAuthenticationPlugin> > <users> > <authenticationUser username="admin" password="******" > groups="users,admins"/> > <authenticationUser username="sharkTool" password="********" > groups="serverTools"/> > <authenticationUser username="adminConsole" > password="*********" > groups="serverTools"/> > </users> > </simpleAuthenticationPlugin> > <authorizationPlugin> > <map> > <authorizationMap> > <authorizationEntries> > <authorizationEntry queue=">" read="admins" write="admins" > admin="admins" /> > <authorizationEntry queue="WF.>" read="users,serverTools" > write="serverTools" admin="admins,serverTools" /> > > <authorizationEntry topic=">" read="admins" write="admins" > admin="admins" /> > <authorizationEntry topic="WF.>" read="users,serverTools" > write="serverTools" admin="admins,serverTools" /> > > <authorizationEntry topic="ActiveMQ.Advisory.>" > read="guests,users,serverTools" write="guests,users,serverTools" > admin="guests,users,serverTools"/> > </authorizationEntries> > > <!-- let's assign roles to temporary destinations. comment > this entry if we don't want any roles assigned to temp destinations > <tempDestinationAuthorizationEntry> > <tempDestinationAuthorizationEntry > read="tempDestinationAdmins" write="tempDestinationAdmins" > admin="tempDestinationAdmins"/> > </tempDestinationAuthorizationEntry> --> > </authorizationMap> > </map> > </authorizationPlugin> > > > </plugins> > <!-- Destination specific policies using destination names or > wildcards --> > <destinationPolicy> > <policyMap> > <policyEntries> > <policyEntry queue=">" memoryLimit="500mb"/> > <policyEntry topic=">" memoryLimit="500mb"> > <!-- you can add other policies too such as these > <dispatchPolicy> > <strictOrderDispatchPolicy/> > </dispatchPolicy> > <subscriptionRecoveryPolicy> > <lastImageSubscriptionRecoveryPolicy/> > </subscriptionRecoveryPolicy> > --> > </policyEntry> > </policyEntries> > </policyMap> > </destinationPolicy> > > <!-- Use the following to configure how ActiveMQ is exposed in > JMX --> > <managementContext> > <managementContext createConnector="false"/> > </managementContext> > > <!-- The store and forward broker networks ActiveMQ will listen > to --> > <networkConnectors> > <!-- by default just auto discover the other brokers --> > <!-- <networkConnector name="default-nc" > uri="multicast://default"/>--> > <!-- Example of a static configuration: > <networkConnector name="host1 and host2" > uri="static://(tcp://host1:61616,tcp://host2:61616)"/> > --> > </networkConnectors> > > <persistenceAdapter> > <amqPersistenceAdapter syncOnWrite="false" > directory="${activemq.base}/data" maxFileLength="20 mb"/> > </persistenceAdapter> > > > > <sslContext> > <sslContext keyStore="file:${activemq.base}/conf/broker.ks" > keyStorePassword="password" > trustStore="file:${activemq.base}/conf/broker.ts" > trustStorePassword="password"/> > </sslContext> > > <!-- The maximum about of space the broker will use before > slowing down producers --> > <systemUsage> > <systemUsage> > <memoryUsage> > <memoryUsage limit="20 mb"/> > </memoryUsage> > <storeUsage> > <storeUsage limit="1 gb" name="foo"/> > </storeUsage> > <tempUsage> > <tempUsage limit="100 mb"/> > </tempUsage> > </systemUsage> > </systemUsage> > > > <!-- The transport connectors ActiveMQ will listen to --> > <transportConnectors> > <transportConnector name="openwire" > uri="tcp://localhost:61616"/> > <!-- <transportConnector name="ssl" > uri="ssl://localhost:61617"/> > <transportConnector name="stomp" uri="stomp://localhost:61613"/> > <transportConnector name="xmpp" > uri="xmpp://localhost:61222"/> --> > </transportConnectors> > <destinations> > <queue physicalName="WF.Regraph" /> > <queue physicalName="WF.Notify"/> > <topic physicalName="WF.Events"/> > </destinations> > <destinationInterceptors> > <virtualDestinationInterceptor> > <virtualDestinations> > <compositeQueue name="WF.IncomingEvents"> > <forwardTo> > <queue physicalName="WF.Regraph"/> > <queue physicalName="WF.Notify"/> > <topic physicalName="WF.Events" /> > </forwardTo> > </compositeQueue> > </virtualDestinations> > </virtualDestinationInterceptor> > </destinationInterceptors> > > > </broker> > > > > -- > > > > > > On Fri, Jan 15, 2010 at 10:24 AM, David Delbecq <de...@oma.be > > <mailto:de...@oma.be>> wrote: > > > > Hello, > > > > can someone explain me why my activeMQ client is stalled in such > > situation (i have 10 threads blocked like this having as > > consequence a complete blocking of my client application): > > > > "http-8081-Processor6" daemon prio=10 tid=00841c38 nid=512 > > lwp_id=2426611 waiting on condition [236f3000..236f153c] > > java.lang.Thread.State: WAITING (parking) > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for <40ae1828> (a > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > > at > > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1889) > > at > > > java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317) > > at > > > org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40) > > at > > > org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:80) > > at > > > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1233) > > at > > org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1659) > > - locked <40ae01e8> (a java.lang.Object) > > at > > > org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:227) > > at > > > org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:241) > > at > > be.rmi.shark.JMSEventManager.sendEvent(JMSEventManager.java:146) > > > > > > considering that the activeMQ lists size, as referenced by > > activeMQ admin interface are as follow: > > Name Number Of Pending Messages Number Of Consumers > > Messages Sent Messages Received Views Operations > > WF.Regraph 0 1 1714 1720 Browse Send To > Purge Delete > > WF.Notify 0 1 1696 1742 Browse Send To > Purge Delete > > WF.IncomingEvents 0 0 0 0 Browse > Send To Purge Delete > > > > > > Is this to be considered normal that i must restart activeMQ > > producers on a regular basis because of those this stalling? Where > > can i find clear documentation on proper configuration of activeMQ > > to prevent this stalling? Thank you. > > > > -- > > David Delbecq > > ICT > > Institut Royal Météorologique > > Ext:557 > > > > > > > > > -- > David Delbecq > ICT > Institut Royal Météorologique > Ext:557 > > -- http://blog.garytully.com Open Source Integration http://fusesource.com
