I downgraded to Stomp-1.0.4 without any progress.
I notice when I browse using the shipped console and using hawtio those
queues that are apparently "stuck", no messages are listed yet the pending
count is dozens.
Could use some guidance to help narrow the scope of this problem down.
On 23 August 2014 10:18, James Green <james.mk.gr...@gmail.com> wrote:
> Usually a restart of amq resolves this issue. Today is different.
>
> After two restarts I see a few dozen messages dequeued then its all stop.
> Nothing in the logs other than DEBUG or INFO. Other servers appear fine.
>
> I can use stomp to send a message on but the script that subscribes and
> dequeues them fails to receive any one the freeze occurs.
>
> Only thing I see different is PECL stomp 1.0.5 on this server and older
> versions elsewhere. Are there know issues here?
>
> Incidentally I checked with hawtio but failed to see anything wrong.
> Memory and file handles in particular well within limits.
>
> Thanks,
>
> James
>
>
> On 21 August 2014 09:49, James Green <james.mk.gr...@gmail.com> wrote:
>
>> One of our spoke machines shows evidence that it is not delivering the
>> messages to consumers as expected. Of the queues in question is named
>> "DeliveryNotifications.Inbound" and I've switched on debug logging to
>> capture what I can.
>>
>> During the following log segment you can see there are messages for
>> delivery (console shows 139) and an automated PHP script comes along via
>> STOMP to collect them, but receives none.
>>
>> 2014-08-21 09:40:31,839 | DEBUG | DeliveryNotifications.Inbound toPageIn:
>> 139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
>> enqueueCount: 142, dequeueCount: 4, memUsage:0 |
>> org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
>> Task-56
>> 2014-08-21 09:40:35,905 | DEBUG | queue://DeliveryNotifications.Inbound
>> expiring messages .. | org.apache.activemq.broker.region.Queue | ActiveMQ
>> Broker[drax] Scheduler
>> 2014-08-21 09:40:35,905 | DEBUG | queue://DeliveryNotifications.Inbound
>> expiring messages done. | org.apache.activemq.broker.region.Queue |
>> ActiveMQ Broker[drax] Scheduler
>> 2014-08-21 09:40:35,905 | DEBUG | DeliveryNotifications.Inbound toPageIn:
>> 139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
>> enqueueCount: 142, dequeueCount: 4, memUsage:0 |
>> org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
>> Task-56
>> 2014-08-21 09:40:38,699 | DEBUG | drax adding consumer:
>> ID:drax-35978-1408600805652-4:9348:-1:1 for destination:
>> queue://DeliveryNotifications.Inbound |
>> org.apache.activemq.broker.region.AbstractRegion | ActiveMQ Transport:
>> tcp:///127.0.0.1:38070@61612
>> 2014-08-21 09:40:38,700 | DEBUG | queue://DeliveryNotifications.Inbound
>> add sub: 4, dequeues: 4, dispatched: 0, inflight: {} |
>> org.apache.activemq.broker.region.Queue | ActiveMQ Transport:
>> tcp:///127.0.0.1:38070@61612
>> 2014-08-21 09:40:38,700 | DEBUG | DeliveryNotifications.Inbound toPageIn:
>> 139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
>> enqueueCount: 142, dequeueCount: 4, memUsage:0 |
>> org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
>> Task-56
>> 2014-08-21 09:41:05,905 | DEBUG | queue://DeliveryNotifications.Inbound
>> expiring messages .. | org.apache.activemq.broker.region.Queue | ActiveMQ
>> Broker[drax] Scheduler
>> 2014-08-21 09:41:05,905 | DEBUG | queue://DeliveryNotifications.Inbound
>> expiring messages done. | org.apache.activemq.broker.region.Queue |
>> ActiveMQ Broker[drax] Scheduler
>> 2014-08-21 09:41:05,905 | DEBUG | DeliveryNotifications.Inbound toPageIn:
>> 139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
>> enqueueCount: 142, dequeueCount: 4, memUsage:0 |
>> org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
>> Task-53
>> 2014-08-21 09:41:35,905 | DEBUG | queue://DeliveryNotifications.Inbound
>> expiring messages .. | org.apache.activemq.broker.region.Queue | ActiveMQ
>> Broker[drax] Scheduler
>> 2014-08-21 09:41:35,905 | DEBUG | queue://DeliveryNotifications.Inbound
>> expiring messages done. | org.apache.activemq.broker.region.Queue |
>> ActiveMQ Broker[drax] Scheduler
>> 2014-08-21 09:41:35,905 | DEBUG | DeliveryNotifications.Inbound toPageIn:
>> 139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
>> enqueueCount: 142, dequeueCount: 4, memUsage:0 |
>> org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
>> Task-53
>> 2014-08-21 09:42:05,906 | DEBUG | queue://DeliveryNotifications.Inbound
>> expiring messages .. | org.apache.activemq.broker.region.Queue | ActiveMQ
>> Broker[drax] Scheduler
>> 2014-08-21 09:42:05,906 | DEBUG | queue://DeliveryNotifications.Inbound
>> expiring messages done. | org.apache.activemq.broker.region.Queue |
>> ActiveMQ Broker[drax] Scheduler
>> 2014-08-21 09:42:05,906 | DEBUG | DeliveryNotifications.Inbound toPageIn:
>> 139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
>> enqueueCount: 142, dequeueCount: 4, memUsage:0 |
>> org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
>> Task-57
>> 2014-08-21 09:42:18,775 | DEBUG | Unregistering MBean
>> org.apache.activemq:type=Broker,brokerName=drax,destinationType=Queue,destinationName=DeliveryNotifications.Inbound,endpoint=Consumer,clientId=ID_drax-35978-1408600805652-4_9348,consumerId=ID_drax-35978-1408600805652-4_9348_-1_1
>> | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ Transport:
>> tcp:///127.0.0.1:38070@61612
>> 2014-08-21 09:42:18,775 | DEBUG | drax removing consumer:
>> ID:drax-35978-1408600805652-4:9348:-1:1 for destination:
>> queue://DeliveryNotifications.Inbound |
>> org.apache.activemq.broker.region.AbstractRegion | ActiveMQ Transport:
>> tcp:///127.0.0.1:38070@61612
>> 2014-08-21 09:42:18,775 | DEBUG | queue://DeliveryNotifications.Inbound
>> remove sub: QueueSubscription:
>> consumer=ID:drax-35978-1408600805652-4:9348:-1:1, destinations=1,
>> dispatched=0, delivered=0, pending=0, lastDeliveredSeqId: 0, dequeues: 4,
>> dispatched: 4, inflight: 0 | org.apache.activemq.broker.region.Queue |
>> ActiveMQ Transport: tcp:///127.0.0.1:38070@61612
>> 2014-08-21 09:42:18,776 | DEBUG | DeliveryNotifications.Inbound toPageIn:
>> 139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
>> enqueueCount: 142, dequeueCount: 4, memUsage:0 |
>> org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
>> Task-57
>> 2014-08-21 09:42:27,943 | DEBUG | drax adding consumer:
>> ID:drax-35978-1408600805652-4:9368:-1:1 for destination:
>> queue://DeliveryNotifications.Inbound |
>> org.apache.activemq.broker.region.AbstractRegion | ActiveMQ Transport:
>> tcp:///127.0.0.1:38178@61612
>> 2014-08-21 09:42:27,944 | DEBUG | queue://DeliveryNotifications.Inbound
>> add sub: 4, dequeues: 4, dispatched: 0, inflight: {} |
>> org.apache.activemq.broker.region.Queue | ActiveMQ Transport:
>> tcp:///127.0.0.1:38178@61612
>> 2014-08-21 09:42:27,944 | DEBUG | DeliveryNotifications.Inbound toPageIn:
>> 139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
>> enqueueCount: 142, dequeueCount: 4, memUsage:0 |
>> org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
>> Task-56
>> 2014-08-21 09:42:35,906 | DEBUG | queue://DeliveryNotifications.Inbound
>> expiring messages .. | org.apache.activemq.broker.region.Queue | ActiveMQ
>> Broker[drax] Scheduler
>> 2014-08-21 09:42:35,907 | DEBUG | queue://DeliveryNotifications.Inbound
>> expiring messages done. | org.apache.activemq.broker.region.Queue |
>> ActiveMQ Broker[drax] Scheduler
>> 2014-08-21 09:42:35,907 | DEBUG | DeliveryNotifications.Inbound toPageIn:
>> 139, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
>> enqueueCount: 142, dequeueCount: 4, memUsage:0 |
>> org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[drax]
>> Task-56
>>
>> Any ideas what's happening here? Or what we can do to debug further?
>>
>> Thanks,
>>
>> James
>>
>>
>
