One of our spoke machines shows evidence that it is not delivering the
messages to consumers as expected. Of the queues in question is named
"DeliveryNotifications.Inbound" and I've switched on debug logging to
capture what I can.
During the following log segment you can see there are messages for
delivery (console shows 139) and an automated PHP script comes along via
STOMP to collect them, but receives none.
2014-08-21 09:40:31,839 | DEBUG | DeliveryNotifications.Inbound toPageIn:
139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
enqueueCount: 142, dequeueCount: 4, memUsage:0 |
org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
Task-56
2014-08-21 09:40:35,905 | DEBUG | queue://DeliveryNotifications.Inbound
expiring messages .. | org.apache.activemq.broker.region.Queue | ActiveMQ
Broker[drax] Scheduler
2014-08-21 09:40:35,905 | DEBUG | queue://DeliveryNotifications.Inbound
expiring messages done. | org.apache.activemq.broker.region.Queue |
ActiveMQ Broker[drax] Scheduler
2014-08-21 09:40:35,905 | DEBUG | DeliveryNotifications.Inbound toPageIn:
139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
enqueueCount: 142, dequeueCount: 4, memUsage:0 |
org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
Task-56
2014-08-21 09:40:38,699 | DEBUG | drax adding consumer:
ID:drax-35978-1408600805652-4:9348:-1:1 for destination:
queue://DeliveryNotifications.Inbound |
org.apache.activemq.broker.region.AbstractRegion | ActiveMQ Transport:
tcp:///127.0.0.1:38070@61612
2014-08-21 09:40:38,700 | DEBUG | queue://DeliveryNotifications.Inbound add
sub: 4, dequeues: 4, dispatched: 0, inflight: {} |
org.apache.activemq.broker.region.Queue | ActiveMQ Transport:
tcp:///127.0.0.1:38070@61612
2014-08-21 09:40:38,700 | DEBUG | DeliveryNotifications.Inbound toPageIn:
139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
enqueueCount: 142, dequeueCount: 4, memUsage:0 |
org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
Task-56
2014-08-21 09:41:05,905 | DEBUG | queue://DeliveryNotifications.Inbound
expiring messages .. | org.apache.activemq.broker.region.Queue | ActiveMQ
Broker[drax] Scheduler
2014-08-21 09:41:05,905 | DEBUG | queue://DeliveryNotifications.Inbound
expiring messages done. | org.apache.activemq.broker.region.Queue |
ActiveMQ Broker[drax] Scheduler
2014-08-21 09:41:05,905 | DEBUG | DeliveryNotifications.Inbound toPageIn:
139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
enqueueCount: 142, dequeueCount: 4, memUsage:0 |
org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
Task-53
2014-08-21 09:41:35,905 | DEBUG | queue://DeliveryNotifications.Inbound
expiring messages .. | org.apache.activemq.broker.region.Queue | ActiveMQ
Broker[drax] Scheduler
2014-08-21 09:41:35,905 | DEBUG | queue://DeliveryNotifications.Inbound
expiring messages done. | org.apache.activemq.broker.region.Queue |
ActiveMQ Broker[drax] Scheduler
2014-08-21 09:41:35,905 | DEBUG | DeliveryNotifications.Inbound toPageIn:
139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
enqueueCount: 142, dequeueCount: 4, memUsage:0 |
org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
Task-53
2014-08-21 09:42:05,906 | DEBUG | queue://DeliveryNotifications.Inbound
expiring messages .. | org.apache.activemq.broker.region.Queue | ActiveMQ
Broker[drax] Scheduler
2014-08-21 09:42:05,906 | DEBUG | queue://DeliveryNotifications.Inbound
expiring messages done. | org.apache.activemq.broker.region.Queue |
ActiveMQ Broker[drax] Scheduler
2014-08-21 09:42:05,906 | DEBUG | DeliveryNotifications.Inbound toPageIn:
139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
enqueueCount: 142, dequeueCount: 4, memUsage:0 |
org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
Task-57
2014-08-21 09:42:18,775 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=drax,destinationType=Queue,destinationName=DeliveryNotifications.Inbound,endpoint=Consumer,clientId=ID_drax-35978-1408600805652-4_9348,consumerId=ID_drax-35978-1408600805652-4_9348_-1_1
| org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ Transport:
tcp:///127.0.0.1:38070@61612
2014-08-21 09:42:18,775 | DEBUG | drax removing consumer:
ID:drax-35978-1408600805652-4:9348:-1:1 for destination:
queue://DeliveryNotifications.Inbound |
org.apache.activemq.broker.region.AbstractRegion | ActiveMQ Transport:
tcp:///127.0.0.1:38070@61612
2014-08-21 09:42:18,775 | DEBUG | queue://DeliveryNotifications.Inbound
remove sub: QueueSubscription:
consumer=ID:drax-35978-1408600805652-4:9348:-1:1, destinations=1,
dispatched=0, delivered=0, pending=0, lastDeliveredSeqId: 0, dequeues: 4,
dispatched: 4, inflight: 0 | org.apache.activemq.broker.region.Queue |
ActiveMQ Transport: tcp:///127.0.0.1:38070@61612
2014-08-21 09:42:18,776 | DEBUG | DeliveryNotifications.Inbound toPageIn:
139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
enqueueCount: 142, dequeueCount: 4, memUsage:0 |
org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
Task-57
2014-08-21 09:42:27,943 | DEBUG | drax adding consumer:
ID:drax-35978-1408600805652-4:9368:-1:1 for destination:
queue://DeliveryNotifications.Inbound |
org.apache.activemq.broker.region.AbstractRegion | ActiveMQ Transport:
tcp:///127.0.0.1:38178@61612
2014-08-21 09:42:27,944 | DEBUG | queue://DeliveryNotifications.Inbound add
sub: 4, dequeues: 4, dispatched: 0, inflight: {} |
org.apache.activemq.broker.region.Queue | ActiveMQ Transport:
tcp:///127.0.0.1:38178@61612
2014-08-21 09:42:27,944 | DEBUG | DeliveryNotifications.Inbound toPageIn:
139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
enqueueCount: 142, dequeueCount: 4, memUsage:0 |
org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
Task-56
2014-08-21 09:42:35,906 | DEBUG | queue://DeliveryNotifications.Inbound
expiring messages .. | org.apache.activemq.broker.region.Queue | ActiveMQ
Broker[drax] Scheduler
2014-08-21 09:42:35,907 | DEBUG | queue://DeliveryNotifications.Inbound
expiring messages done. | org.apache.activemq.broker.region.Queue |
ActiveMQ Broker[drax] Scheduler
2014-08-21 09:42:35,907 | DEBUG | DeliveryNotifications.Inbound toPageIn:
139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
enqueueCount: 142, dequeueCount: 4, memUsage:0 |
org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
Task-56
Any ideas what's happening here? Or what we can do to debug further?
Thanks,
James
