The default limit is indeed 1024 message groups per destination.
However you increase that by adding the following configuration to your
broker's activemq.xml:
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">">
<messageGroupMapFactory>
<cachedMessageGroupMessageFactory cacheSize="2048"/>
</messageGroupMapFactory>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
In addition to cachedMessageGroupMessageFactory (default) there's also the
messageGroupHashBucketFactory and simpleMessageGroupMapFactory factories.
Just dig through the source code. They're all in the
org.apache.activemq.broker.region.group package.
However, it's not advisable to simply increase the cache size to allow for
millions of entries, not least because there's no real way of knowing the
upper limit of how many you'll need.
Therefore you're better off calculating a poor man's hash of the underlying
value mod 1024 to calculate the group id. With such a large many-to-one
mapping many values will map to the same group id. However, that doesn't
matter. As long as the hash of the value is the same each time it's taken
that's all that matters. The groups are only needed to divide the work up
amongst your consumers. Exactly how many values map to each group id is
irrelevant.
Example: an input value of 111-22-1234 might hash to 0xabcd1234. Use that
hash value to calculate the group id:
0xabcd1234 % 1024 = group id
The only thing one has to watch for is ensuring that the distribution of
hash values to group id is uniform. Otherwise you'll end up with consumers
handling much more workload than their fair share.
It's worth noting that if you plan to use message group sequence numbers
(for whatever reason) that you'll have to increment it's value yourself as
the broker doesn't manage the sequence id by default.
I hope this makes sense.
Thanks,
Paul
On Mon, Jan 25, 2016 at 4:55 PM, derek_mlist <ml...@mckittrick.us> wrote:
>
> Hello,
>
> We are investigating using message groups as a technique to insure "in
> order" processing of various IDs. In the past (2010), I have found posts
> mentioning that more than 1024 message groups will trigger an overflow for
> the default implementations.
> http://scott.cranton.com/2010/09/activemq-message-groups.html
>
> I can't find anything specific that is newer on the topic except "there
> could be a massive number of individual message groups we use hash buckets
> rather than the actual JMSXGroupID string." The word "massive" is
> ambiguous and relative. Say we are using social security numbers as our
> IDs and have millions of customers. Is 2 million records going to be
> supportable by default?
>
> I also understand hashmaps pretty well. I don't understand what is meant
> by using "hash buckets" rather than the JMSXGroupID string. In a usual
> hashmap implementation a bucket would store a certain number of hashed
> keys.
> What does it matter if the key is stored or a hashed key is stored, and
> what
> is the relation to the bucket?
>
> Thanks in advance.
>
> Derek
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Message-Group-Limitations-how-many-simulataneous-groups-are-supported-tp4706412.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
