I'm doing performance characterization of ActiveMQ when a network of brokers runs across a high-latency (100ms range) WAN. When my producer on one side of the WAN sends faster than our meager allocation of the WAN's bandwidth, I quickly see all messages fail to be delivered to the end consumer.
These are the three critical elements of the problem, which all have to be present for it to happen: 1. Messages have a TTL set (the same for all messages), so they'll eventually expire. We're using Camel to do this for us, but it would be the same if it were set directly without Camel's help. 2. Producers are sending messages faster (in aggregate) than our bandwidth allocation on the WAN. This means we're guaranteed to not deliver some of the messages to the end consumer, but in practice we're not delivering any of them. 3. There is a non-trivial amount of latency across the WAN. As messages are sent, they begin queuing on the sender-side broker. As time goes on, the messages that are still in the producer-side broker's message store get closer and closer to expiring, until eventually the message at the head of the message store is within the WAN's latency value (e.g. 100ms) of the message's expiration time. The amount of time it takes for this to happen depends on how long it takes messages to time out and on the difference between the producer's send rate and the WAN's bandwidth, but it will eventually happen. This message will be sent by the producer-side broker (because although it's really close to expiring, it hasn't expired yet), but when it's received by the consumer-side broker, an amount of time equal to the WAN latency has passed, so it's expired and gets discarded by the consumer-side broker instead of getting delivered to the consumer. >From this point onwards, no messages will get successfully delivered to the consumer. As the messages in the producer-side broker's message store get closer to and eventually reach their expiration times, each message will either be within the WAN latency of its timeout or after its timeout. If the former, it will get sent across the WAN but discarded by the consumer-side broker; if the latter, it will get discarded by the producer-side broker and that broker will find the next message in the message store that isn't yet expired (but will be by the time it arrives) and send it instead. As a result, all messages from that point onward either expire on the producer-side broker or the consumer-side broker. Even though there are lots of messages in the producer-side broker's message store that could be delivered successfully, ActiveMQ instead sends the first message in the message store even though an outside observer knows it will just get thrown away. Ideally, ActiveMQ should prioritize messages that are expected to reach an end consumer over ones that are expected to time out before they get there, to minimize wasteful use of scarce resources such as network links. Doing that automatically and without any the user having to provide lots of up-front configuration of network topology sounds hard, particularly when considering that network link performance can vary over time and that different consumers may have different network paths from the producer to the consumer. But I think it would be very useful to have a setting that allows a user to specify that messages within X milliseconds of their expiration time will be discarded by the broker rather than forwarded to the next broker. The default should be 0 (so all messages that haven't actually expired would be forwarded), but if I know that my network path has a certain latency, I should be able to configure the broker to not even try delivering messages that I know aren't likely to make it to an end consumer, so that messages that will can be sent instead. Does this seem like a reasonable feature to add? If so, I'll submit a JIRA for it. Tim
