On 2/5/07, GaryG <[EMAIL PROTECTED]> wrote:
GaryG wrote:
>
>
> On 2/5/07, Givental, Gary wrote:
>> Ah yes, I should have clarified that :)
>>
>> The nature of the applications makes that impossible. We need to
>> ensure that A) all the tasks are processed in sequential order, B)
>> each work task is tied to a specific business entity. If 2 tasks need
>> to be done for the same business entity, they *must* be processed in
>> time-sequential order, and be processed by the Worker.
>>
>> These two requirements do not allow us to use a Queue for all the
>> worker tasks, since that would mean they they might be processed out
>> of order, and even worse, 2 tasks for the same business entity might
>> be processed out of time-sequence.
>
>
>
> Aha. Looks like you need to use Message Groups. Using the business entity
> as the JMSXGroupID value - then you get load balancing across your
> business entities and workers - yet each business entity's messages are
> processed in time-sequential order
>
> http://activemq.apache.org/message-groups.html
>
> If you need to preserve the order of all messages across all business
> entities (i.e. process all messages essentially single threaded), then you
> could use Exclusive Consumers
>
> http://activemq.apache.org/exclusive-consumer.html
>
> in either case, the broker does the hard work of tracking workers and load
> balancing etc
> --
> James
>
In response to this, James, does it effect performance negatively to have
thousands of message groups? on the order of 10,000?
No, not really. We kinda guessed folks would use lots of message
groups, so we end up using the hash function of the JMSXGroupID
string, then assign hash bucket ranges to consumers. So you can use as
many message groups as you like. (If you're interested, this code is
in the MessageGroupHashBucket). The default number of buckets is 1024
which should be fine for most uses but you can make that higher if you
require - the only real cost of an increased number of buckets is more
RAM for the data structure.
Also, it sounds like Message Groups are Topics (publish/subscribe) is that
correct?
No, they are queues (each message is processed by exactly one
consumer). Message Groups kinda implement sticky load balancing like
websites - only rather than a jsessionid or cookie, its using
JMSXGroupId as the 'session id'. Another way to look at Message Groups
is they are a way of splitting a queue into many separate Exclusive
Queues which are then spread across the consumers.
What about the use case (building on the Stock Symbols example on the
activemq site) where perhaps all the Stock Symbols (MSFT, IBM, etc) are
unique Message Groups, but we also want to group all the stock that are for
Internet Companies, and a group for all the automotive industry companies,
etc...is there sub-grouping in other words?
No, I'm afraid not. There is just one level of grouping, the value of
the JMSXGroupID you choose. So if you want to process all automotive
companies in order by one single thread / consumer - then choose, say,
"automotive" as the JMSXGroupID.
You could do futher traffic shaping by using multiple queues. So you
could use a queue for "automotive" then use message groups within it
if needed.
I'm also concerned about memory requirements and performance of the broker
itself
It doesn't affect memory & performance much, as the broker already
load balances messages on a queue across the consumers; its really
just an extra check to see which consumer owns a group using an
efficient hash bucket data structure.
--
James
-------
http://radio.weblogs.com/0112098/
