On Friday 05 September 2008, Bryan Murphy said: > Sounds like a nightmare to me. Maybe I don't understand what you're trying > to accomplish, but I think something like Hadoop would be a better fit for > this kind of problem.
sounds like a job that JINI was designed for. > > Bryan > > On Fri, Sep 5, 2008 at 12:10 PM, Martin_Cornelius < > > [EMAIL PROTECTED]> wrote: > > Hi all, > > > > i am currently evaluating the scalability of Networks of ActiveMQ Brokers > > with a large number of brokers. The actual drive behind this evaluation: > > I have to design a distributed system that is completely serverless. It > > shall be possible to switch of any of the machines in the system without > > alloying the distributed functionality. The number of active machines in > > the system might vary from only 2 to about 100. > > > > My current idea: Have an (presumably embedded) ActiveMQ Broker running on > > any of the machines, and let all this Brokers form a 'Network of Brokers' > > (abbreviated NOB in the follwing) > > > > So far, i found that in an NOB a message is forwarded only to those > > brokers where at least one client has subscribed to the topic resp. queue > > the message is sent to. Really fine. If a client subscribes to a queue, > > the broker where the subscription is made sends a single short > > control-message to all other brokers of the NOB. That would still allow > > for thousands of brokers, fine again. > > > > However, i just realized one fact that may compromise scalabilty: If a > > client subscribes to a topic, the broker where the subscription is done > > sends a not-so-short control-message to all other brokers. So far not too > > bad, but now comes the problem: Every broker seems to react on this > > control-message by sending another not-so-short control-message to all > > other > > brokers. This means, if N brokers are present, every subscription to a > > topic > > results in about (N-1)*N*2 control-messages (The additional factor two > > comes > > from the fact, that the final recipient-broker seems to send another > > response control-message). BTW, The length of the control-messages being > > sent seems to be closely related to the string length of the topic name. > > > > I have verified this behaviour with only up to 6 brokers, so i'm not > > totally > > shure if it will extrapolate to a larger number, but i suspect it does. > > This > > would mean, that with e.g. 100 Brokers every subscription to a topic > > would result in 100*99*2 control messages being sent. If a single control > > message has about 400 bytes (what i observed), this means a single > > subscription to a > > topic would result in about 8 Megabyte network traffic. This might even > > be sustainable, but the quadratic order kills scalabilty -- with 1000 > > Brokers we would have to transfer about 1 Gigabyte, what is no longer > > feasible. > > > > Finally, my questions: Is there any configuration option by which this > > quadratic order can be avoided, is my idea simply foolish, or are my > > conclusions wrong ? Note that i gained my conclusions more or less > > 'empirically' using wireshark. > > > > thanks for any advice, Martin > > > > -- > > View this message in context: > > http://www.nabble.com/Scalability-of-%27Networks-of-Brokers%27-tp19335699 > >p19335699.html Sent from the ActiveMQ - User mailing list archive at > > Nabble.com. -- Card #1. Uncle Bob’s Three Laws • Write no production code except to pass a failing test. • Write only enough of a test to demonstrate a failure. • Write only enough production code to pass the test.
