We are facing a strange issue while syncing messages from source to target
broker cluster using Core bridge. Our source and target brokers are symmetric
cluster and each has 2 nodes.
Since both source and target cluster has 2 nodes each, we have defined bridges
from source cluster to target cluster as following. Idea is any source node can
send messages to any available target node independently.
This configuration is working fine most of the time but all of sudden we see
some messages in delivering count on any of the source nodes. Bridge stat on
that source node also shows a gap b/w 'Message Acknowledged' vs 'Message
Pending Acknowledgment'. This gap is same as no. of messages showing in
delivering count. Address stat on this queue also shows gap b/w 'Message Added'
and 'Message Acknowledged'. This gap is same as no. of messages showing in
delivering count. If this issue is happening due to networking trouble, then
bridge should auto recover in some time. If new messages come on queue, they
are getting processed but the older delivering count messages are never cleared
on its own.
If we stop and start the bridges then most of the time, these delivering
messages are cleared. We are not able to find the root cause behind this
incident. Any input/lead will be very helpful.
Please also note that at the same time, we have other bridges b/w same source
and target clusters on different addresses but we are not facing this challenge
there.
Source node1
<bridge name="FILES.RECORDER.1">
<queue-name>FILES.RECORDER</queue-name>
<forwarding-address>FILES.RECORDER</forwarding-address>
<retry-interval>45000</retry-interval>
<max-retry-interval>45000</max-retry-interval>
<reconnect-attempts>-1</reconnect-attempts>
<static-connectors>
<connector-ref>target-01</connector-ref>
</static-connectors>
</bridge>
<bridge name="FILES.RECORDER.2">
<queue-name>FILES.RECORDER</queue-name>
<forwarding-address>FILES.RECORDER</forwarding-address>
<retry-interval>45000</retry-interval>
<max-retry-interval>45000</max-retry-interval>
<reconnect-attempts>-1</reconnect-attempts>
<static-connectors>
<connector-ref>target-02</connector-ref>
</static-connectors>
</bridge>
Source node2
<bridge name="FILES.RECORDER.1">
<queue-name>FILES.RECORDER</queue-name>
<forwarding-address>FILES.RECORDER</forwarding-address>
<retry-interval>45000</retry-interval>
<max-retry-interval>45000</max-retry-interval>
<reconnect-attempts>-1</reconnect-attempts>
<static-connectors>
<connector-ref>target-01</connector-ref>
</static-connectors>
</bridge>
<bridge name="FILES.RECORDER.2">
<queue-name>FILES.RECORDER</queue-name>
<forwarding-address>FILES.RECORDER</forwarding-address>
<retry-interval>45000</retry-interval>
<max-retry-interval>45000</max-retry-interval>
<reconnect-attempts>-1</reconnect-attempts>
<static-connectors>
<connector-ref>target-02</connector-ref>
</static-connectors>
</bridge>
