Dear Gary, Thanks for your response. I have enabled the logging and sent a message. The consumer sent the message to BrokerB which was forwarded to BrokerC and processed. The response was put back on BrokerC which forwarded the message to BrokerB and later BrokerA. However, I can see lot of "already routed through this broker once" messages in the log in all the brokers till the message expires after 2 minutes. When the response was forwarded to BrokerB from BrokerC where the client was listening (BrokerB), the BrokerB tries to forward the message to BrokerA or BrokerC instead of letting it consume by the consumer. After much hoping around, the message got finally timed out and removed from the queue leading the consumer to wait for 3 minutes and time out.
I'm adding some important logs from three brokers in the order the response was put back and forwarded: Note: BrokerA = ValhallaMQ, BrokerB = MidgardMQ , and BrokerC = JotunheimMQ, and Selector Key/Correlation Id = 7430d0bb-cae4-47b5-b0b9-710b5b79e6d0 *BrokerC:* 2012-10-23 10:03:22,610 | DEBUG | JotunheimMQ Ignoring sub from MidgardMQ, destination queue://PAYMENT_RESPONSE is not permiited :ConsumerInfo {commandId = 5, responseRequired = true, consumerId = ID:thor.amt.com-40176-1351011796558-1:7:1:1, destination = queue://PAYMENT_RESPONSE, prefetchSize = 1000, maximumPendingMessageLimit = 0, browser = false, dispatchAsync = true, selector = PAYMENT_REQUEST_KEY = '7430d0bb-cae4-47b5-b0b9-710b5b79e6d0', subscriptionName = null, noLocal = false, exclusive = false, retroactive = false, priority = 0, brokerPath = null, optimizedAcknowledge = false, noRangeAcks = false, additionalPredicate = null} | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: tcp:///10.0.0.178:61616@46883 2012-10-23 10:03:22,611 | DEBUG | queue://PAYMENT_RESPONSE add sub: QueueSubscription: consumer=ID:jotunheim.amt.com-50445-1351010815753-2:2:1:31, destinations=0, dispatched=0, delivered=0, pending=0, dequeues: 41, dispatched: 41, inflight: 0 | org.apache.activemq.broker.region.Queue | ActiveMQ Transport: tcp:///10.0.0.178:61616@46881 2012-10-23 10:03:22,613 | DEBUG | JotunheimMQ Ignoring sub from ValhallaMQ, destination queue://PAYMENT_RESPONSE is not permiited :ConsumerInfo {commandId = 96, responseRequired = false, consumerId = ID:valhalla.amt.com-47978-1351010795198-23:2:1:42, destination = queue://PAYMENT_RESPONSE, prefetchSize = 1000, maximumPendingMessageLimit = 0, browser = false, dispatchAsync = true, selector = PAYMENT_REQUEST_KEY = '7430d0bb-cae4-47b5-b0b9-710b5b79e6d0', subscriptionName = null, noLocal = false, exclusive = false, retroactive = false, priority = 0, brokerPath = [ID:midgard.amt.com-34058-1351010814364-0:1], optimizedAcknowledge = false, noRangeAcks = false, additionalPredicate = org.apache.activemq.command.NetworkBridgeFilter@1261824e} | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: tcp:///10.0.0.177:61616@41995 2012-10-23 10:03:22,613 | DEBUG | JotunheimMQ bridged sub on vm://JotunheimMQ#0 from MidgardMQ : ConsumerInfo {commandId = 5, responseRequired = true, consumerId = ID:thor.amt.com-40176-1351011796558-1:7:1:1, destination = queue://PAYMENT_RESPONSE, prefetchSize = 1000, maximumPendingMessageLimit = 0, browser = false, dispatchAsync = true, selector = PAYMENT_REQUEST_KEY = '7430d0bb-cae4-47b5-b0b9-710b5b79e6d0', subscriptionName = null, noLocal = false, exclusive = false, retroactive = false, priority = 0, brokerPath = null, optimizedAcknowledge = false, noRangeAcks = false, additionalPredicate = null} | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: tcp:///10.0.0.178:61616@46881 2012-10-23 10:03:22,613 | DEBUG | JotunheimMQ adding consumer: ID:jotunheim.amt.com-50445-1351010815753-3:2:1:37 for destination: queue://PAYMENT_RESPONSE | org.apache.activemq.broker.region.AbstractRegion | ActiveMQ Transport: tcp:///10.0.0.177:61616@41993 2012-10-23 10:03:22,623 | DEBUG | JotunheimMQ bridged sub on vm://JotunheimMQ#2 from ValhallaMQ : ConsumerInfo {commandId = 96, responseRequired = false, consumerId = ID:valhalla.amt.com-47978-1351010795198-23:2:1:42, destination = queue://PAYMENT_RESPONSE, prefetchSize = 1000, maximumPendingMessageLimit = 0, browser = false, dispatchAsync = true, selector = PAYMENT_REQUEST_KEY = '7430d0bb-cae4-47b5-b0b9-710b5b79e6d0', subscriptionName = null, noLocal = false, exclusive = false, retroactive = false, priority = 0, brokerPath = [ID:midgard.amt.com-34058-1351010814364-0:1], optimizedAcknowledge = false, noRangeAcks = false, additionalPredicate = org.apache.activemq.command.NetworkBridgeFilter@3c62de83} | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: tcp:///10.0.0.177:61616@41993 2012-10-23 10:03:23,677 | DEBUG | bridging (JotunheimMQ -> MidgardMQ) ID:frigga.amt.com-46716-1351011227838-1:1:1:1:14, consumer: ID:jotunheim.amt.com-50445-1351010815753-2:2:1:31, destination queue://PAYMENT_RESPONSE, brokerPath: [ID:jotunheim.amt.com-50445-1351010815753-0:1], message: ActiveMQObjectMessage {commandId = 34, responseRequired = true, messageId = ID:frigga.amt.com-46716-1351011227838-1:1:1:1:14, originalDestination = null, originalTransactionId = null, producerId = ID:jotunheim.amt.com-50445-1351010815753-2:1:1:1, destination = queue://PAYMENT_RESPONSE, transactionId = null, expiration = 1351011923666, timestamp = 1351011803666, arrival = 0, brokerInTime = 1351011803674, brokerOutTime = 1351011803677, correlationId = 7430d0bb-cae4-47b5-b0b9-710b5b79e6d0, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@37bc2e0, marshalledProperties = org.apache.activemq.util.ByteSequence@323701e9, dataStructure = null, redeliveryCounter = 0, size = 1779, properties = {PAYMENT_REQUEST_KEY=7430d0bb-cae4-47b5-b0b9-710b5b79e6d0}, readOnlyProperties = false, readOnlyBody = false, droppable = false} | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ BrokerService[JotunheimMQ] Task-28 2012-10-23 10:05:22,627 | DEBUG | JotunheimMQ remove local subscription for remote ID:thor.amt.com-40176-1351011796558-1:7:1:1 | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: tcp:///10.0.0.178:61616@46881 2012-10-23 10:05:22,627 | DEBUG | JotunheimMQ removed sub on vm://JotunheimMQ#0 from MidgardMQ : ConsumerInfo {commandId = 5, responseRequired = true, consumerId = ID:thor.amt.com-40176-1351011796558-1:7:1:1, destination = queue://PAYMENT_RESPONSE, prefetchSize = 1000, maximumPendingMessageLimit = 0, browser = false, dispatchAsync = true, selector = PAYMENT_REQUEST_KEY = '7430d0bb-cae4-47b5-b0b9-710b5b79e6d0', subscriptionName = null, noLocal = false, exclusive = false, retroactive = false, priority = 0, brokerPath = [ID:midgard.amt.com-34058-1351010814364-0:1], optimizedAcknowledge = false, noRangeAcks = false, additionalPredicate = null} | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: tcp:///10.0.0.178:61616@46881 2012-10-23 10:05:22,629 | DEBUG | JotunheimMQ removed sub on vm://JotunheimMQ#2 from ValhallaMQ : ConsumerInfo {commandId = 96, responseRequired = false, consumerId = ID:valhalla.amt.com-47978-1351010795198-23:2:1:42, destination = queue://PAYMENT_RESPONSE, prefetchSize = 1000, maximumPendingMessageLimit = 0, browser = false, dispatchAsync = true, selector = PAYMENT_REQUEST_KEY = '7430d0bb-cae4-47b5-b0b9-710b5b79e6d0', subscriptionName = null, noLocal = false, exclusive = false, retroactive = false, priority = 0, brokerPath = [ID:midgard.amt.com-34058-1351010814364-0:1, ID:valhalla.amt.com-47978-1351010795198-0:1], optimizedAcknowledge = false, noRangeAcks = false, additionalPredicate = null} | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: tcp:///10.0.0.177:61616@41993 *BrokerB:* 2012-10-23 10:03:22,670 | DEBUG | MidgardMQ Ignoring sub from JotunheimMQ, already routed through this broker once : ConsumerInfo {commandId = 81, responseRequired = false, consumerId = ID:jotunheim.amt.com-50445-1351010815753-2:2:1:31, destination = queue://PAYMENT_RESPONSE, prefetchSize = 1000, maximumPendingMessageLimit = 0, browser = false, dispatchAsync = true, selector = PAYMENT_REQUEST_KEY = '7430d0bb-cae4-47b5-b0b9-710b5b79e6d0', subscriptionName = null, noLocal = false, exclusive = false, retroactive = false, priority = 0, brokerPath = [ID:midgard.amt.com-34058-1351010814364-0:1], optimizedAcknowledge = false, noRangeAcks = false, additionalPredicate = org.apache.activemq.command.NetworkBridgeFilter@6309b022} | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: tcp:///10.0.0.179:61616@35360 2012-10-23 10:03:22,670 | DEBUG | MidgardMQ Ignoring sub from ValhallaMQ, already routed through this broker once : ConsumerInfo {commandId = 96, responseRequired = false, consumerId = ID:valhalla.amt.com-47978-1351010795198-23:2:1:42, destination = queue://PAYMENT_RESPONSE, prefetchSize = 1000, maximumPendingMessageLimit = 0, browser = false, dispatchAsync = true, selector = PAYMENT_REQUEST_KEY = '7430d0bb-cae4-47b5-b0b9-710b5b79e6d0', subscriptionName = null, noLocal = false, exclusive = false, retroactive = false, priority = 0, brokerPath = [ID:midgard.amt.com-34058-1351010814364-0:1], optimizedAcknowledge = false, noRangeAcks = false, additionalPredicate = org.apache.activemq.command.NetworkBridgeFilter@110abafa} | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: tcp:///10.0.0.177:61616@60343 2012-10-23 10:03:22,676 | DEBUG | bridging (MidgardMQ -> ValhallaMQ) ID:thor.amt.com-40176-1351011796558-1:7:1:1:1, consumer: ID:midgard.amt.com-34058-1351010814364-3:2:1:2, destination queue://PAYMENT_REQUEST, brokerPath: [ID:midgard.amt.com-34058-1351010814364-0:1], message: ActiveMQObjectMessage {commandId = 6, responseRequired = false, messageId = ID:thor.amt.com-40176-1351011796558-1:7:1:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:midgard.amt.com-34058-1351010814364-3:1:1:1, destination = queue://PAYMENT_REQUEST, transactionId = null, expiration = 1351011892821, timestamp = 1351011802821, arrival = 0, brokerInTime = 1351011802675, brokerOutTime = 1351011802675, correlationId = 7430d0bb-cae4-47b5-b0b9-710b5b79e6d0, replyTo = null, persistent = false, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@313239d8, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 2147, properties = null, readOnlyProperties = false, readOnlyBody = false, droppable = false} | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ BrokerService[MidgardMQ] Task-35 2012-10-23 10:03:22,680 | DEBUG | MidgardMQ Ignoring sub from JotunheimMQ, already routed through this broker once : ConsumerInfo {commandId = 103, responseRequired = false, consumerId = ID:jotunheim.amt.com-50445-1351010815753-3:2:1:37, destination = queue://PAYMENT_RESPONSE, prefetchSize = 1000, maximumPendingMessageLimit = 0, browser = false, dispatchAsync = true, selector = PAYMENT_REQUEST_KEY = '7430d0bb-cae4-47b5-b0b9-710b5b79e6d0', subscriptionName = null, noLocal = false, exclusive = false, retroactive = false, priority = 0, brokerPath = [ID:midgard.amt.com-34058-1351010814364-0:1, ID:valhalla.amt.com-47978-1351010795198-0:1], optimizedAcknowledge = false, noRangeAcks = false, additionalPredicate = org.apache.activemq.command.NetworkBridgeFilter@126c0233} | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: tcp:///10.0.0.179:61616@35360 2012-10-23 10:03:23,740 | DEBUG | bridging (MidgardMQ -> ValhallaMQ) ID:frigga.amt.com-46716-1351011227838-1:1:1:1:14, consumer: ID:midgard.amt.com-34058-1351010814364-3:2:1:43, destination queue://PAYMENT_RESPONSE, brokerPath: [ID:jotunheim.amt.com-50445-1351010815753-0:1, ID:midgard.amt.com-34058-1351010814364-0:1], message: ActiveMQObjectMessage {commandId = 29, responseRequired = true, messageId = ID:frigga.amt.com-46716-1351011227838-1:1:1:1:14, originalDestination = null, originalTransactionId = null, producerId = ID:midgard.amt.com-34058-1351010814364-3:1:1:1, destination = queue://PAYMENT_RESPONSE, transactionId = null, expiration = 1351011923666, timestamp = 1351011803666, arrival = 0, brokerInTime = 1351011803736, brokerOutTime = 1351011803739, correlationId = 7430d0bb-cae4-47b5-b0b9-710b5b79e6d0, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@522f3bfa, marshalledProperties = org.apache.activemq.util.ByteSequence@1b0ac43d, dataStructure = null, redeliveryCounter = 0, size = 1779, properties = {PAYMENT_REQUEST_KEY=7430d0bb-cae4-47b5-b0b9-710b5b79e6d0}, readOnlyProperties = false, readOnlyBody = false, droppable = false} | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ BrokerService[MidgardMQ] Task-32 *BrokerA:* 2012-10-23 10:03:22,712 | DEBUG | ValhallaMQ bridged sub on vm://ValhallaMQ#42 from MidgardMQ : ConsumerInfo {commandId = 5, responseRequired = true, consumerId = ID:thor.amt.com-40176-1351011796558-1:7:1:1, destination = queue://PAYMENT_RESPONSE, prefetchSize = 1000, maximumPendingMessageLimit = 0, browser = false, dispatchAsync = true, selector = PAYMENT_REQUEST_KEY = '7430d0bb-cae4-47b5-b0b9-710b5b79e6d0', subscriptionName = null, noLocal = false, exclusive = false, retroactive = false, priority = 0, brokerPath = null, optimizedAcknowledge = false, noRangeAcks = false, additionalPredicate = null} | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: tcp:///10.0.0.178:61616@55531 2012-10-23 10:03:22,713 | DEBUG | ValhallaMQ Ignoring sub from JotunheimMQ as already subscribed to matching destination : ConsumerInfo {commandId = 81, responseRequired = false, consumerId = ID:jotunheim.amt.com-50445-1351010815753-2:2:1:31, destination = queue://PAYMENT_RESPONSE, prefetchSize = 1000, maximumPendingMessageLimit = 0, browser = false, dispatchAsync = true, selector = PAYMENT_REQUEST_KEY = '7430d0bb-cae4-47b5-b0b9-710b5b79e6d0', subscriptionName = null, noLocal = false, exclusive = false, retroactive = false, priority = 0, brokerPath = [ID:midgard.amt.com-34058-1351010814364-0:1], optimizedAcknowledge = false, noRangeAcks = false, additionalPredicate = org.apache.activemq.command.NetworkBridgeFilter@352096f2} | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: tcp:///10.0.0.179:61616@40611 2012-10-23 10:03:22,717 | DEBUG | ValhallaMQ bridged sub on vm://ValhallaMQ#46 from JotunheimMQ : ConsumerInfo {commandId = 81, responseRequired = false, consumerId = ID:jotunheim.amt.com-50445-1351010815753-2:2:1:31, destination = queue://PAYMENT_RESPONSE, prefetchSize = 1000, maximumPendingMessageLimit = 0, browser = false, dispatchAsync = true, selector = PAYMENT_REQUEST_KEY = '7430d0bb-cae4-47b5-b0b9-710b5b79e6d0', subscriptionName = null, noLocal = false, exclusive = false, retroactive = false, priority = 0, brokerPath = [ID:midgard.amt.com-34058-1351010814364-0:1], optimizedAcknowledge = false, noRangeAcks = false, additionalPredicate = org.apache.activemq.command.NetworkBridgeFilter@3b29eba3} | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: tcp:///10.0.0.179:61616@40612 2012-10-23 10:03:22,722 | DEBUG | ValhallaMQ Ignoring sub from JotunheimMQ, already routed through this broker once : ConsumerInfo {commandId = 103, responseRequired = false, consumerId = ID:jotunheim.amt.com-50445-1351010815753-3:2:1:37, destination = queue://PAYMENT_RESPONSE, prefetchSize = 1000, maximumPendingMessageLimit = 0, browser = false, dispatchAsync = true, selector = PAYMENT_REQUEST_KEY = '7430d0bb-cae4-47b5-b0b9-710b5b79e6d0', subscriptionName = null, noLocal = false, exclusive = false, retroactive = false, priority = 0, brokerPath = [ID:midgard.amt.com-34058-1351010814364-0:1, ID:valhalla.amt.com-47978-1351010795198-0:1], optimizedAcknowledge = false, noRangeAcks = false, additionalPredicate = org.apache.activemq.command.NetworkBridgeFilter@4f4dfa6a} | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: tcp:///10.0.0.179:61616@40611 2012-10-23 10:05:22,727 | DEBUG | ValhallaMQ removed sub on vm://ValhallaMQ#42 from MidgardMQ : ConsumerInfo {commandId = 5, responseRequired = true, consumerId = ID:thor.amt.com-40176-1351011796558-1:7:1:1, destination = queue://PAYMENT_RESPONSE, prefetchSize = 1000, maximumPendingMessageLimit = 0, browser = false, dispatchAsync = true, selector = PAYMENT_REQUEST_KEY = '7430d0bb-cae4-47b5-b0b9-710b5b79e6d0', subscriptionName = null, noLocal = false, exclusive = false, retroactive = false, priority = 0, brokerPath = [ID:midgard.amt.com-34058-1351010814364-0:1], optimizedAcknowledge = false, noRangeAcks = false, additionalPredicate = null} | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: tcp:///10.0.0.178:61616@55531 2012-10-23 10:05:22,732 | DEBUG | ValhallaMQ removed sub on vm://ValhallaMQ#46 from JotunheimMQ : ConsumerInfo {commandId = 81, responseRequired = false, consumerId = ID:jotunheim.amt.com-50445-1351010815753-2:2:1:31, destination = queue://PAYMENT_RESPONSE, prefetchSize = 1000, maximumPendingMessageLimit = 0, browser = false, dispatchAsync = true, selector = PAYMENT_REQUEST_KEY = '7430d0bb-cae4-47b5-b0b9-710b5b79e6d0', subscriptionName = null, noLocal = false, exclusive = false, retroactive = false, priority = 0, brokerPath = [ID:midgard.amt.com-34058-1351010814364-0:1, ID:jotunheim.amt.com-50445-1351010815753-0:1], optimizedAcknowledge = false, noRangeAcks = false, additionalPredicate = null} | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: tcp:///10.0.0.179:61616@40612 Thanks, Raghu -- View this message in context: http://activemq.2283324.n4.nabble.com/Request-Response-Model-Not-Working-in-Network-of-Brokers-Please-Help-tp4658071p4658121.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.