Hello, Well, we've figured this out so I'll record it here for posterity.
Our master/slave setup was incorrectly configured. Originally our architecture was going to have two clusters that talked to each other. In our phased development approach we scaled back to work with just one initially. However, the Network Connector config node was inadvertently/misunderstood and left in: <networkConnector uri="masterslave:(tcp://localhost:61616,tcp://[slavehost]:61616)"/> What caused us to zero in on this was that every time we connected to one of our queues, in the Admin Console we would see our consumers in the "Active Consumers" page for the queue, but ALSO an additional consumer: "NC_[our_master_brokerName]_inbound_[our_master_brokerName]" This consumer would also disappear when our consumers disconnected from the queue. Behavior would manifest as such: For a queue: * Enqueue is always correct, it's the total number sent. * When the consumers are done consuming what they will be sent, the inflights are 0. * The hoarded difference always corresponds to the # of rows in the ACTIVEMQ_MSGS table that won't budge. * Dequeue is also always correct, it's the total number received by the consumers BUT that INCLUDES the NC consumers. Example: We send 150 messages to the queue. * Our client consumer: Enqueue/Dequeue/Dispatch are all 50 each. * The NC consumer: Enqueue/Dequeue/Dispatch are all 50 each. * Pending for queue 50 = 50 rows in the ACTIVEMQ_MSGS db Our client and the NC consumer would alternate consuming messages until 100, and then the last contiguous block of 50 would be hoarded. Oddly enough, a couple of times we left the cluster up, but inactive, all night. And when we returned to run tests, there would be no creation of NC consumers and no hoarding. Everything worked fine. Our tip off was that in this scenario, failover from the master to slave worked fine, so it was clear that the NC was not needed within a single master/slave cluster. [Not sure why this is?] But without fail, if we restarted the cluster and started using it relatively soon, we would see the NC consumers and the hoarding. Whoosh. Hopefully this might help someone else save some time, -h -- View this message in context: http://activemq.2283324.n4.nabble.com/Master-Slave-Message-Hoarding-tp4669591p4670940.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
