Hello Tim. Thanks for taking the time for a response.
You are correct. The setup is being used to keep the 2 datacenters completely in sync. It allows us to switch our reporting frontend to the other datacenter at any time if we are having issues at the opposite datacenter. The messages are consumed from ActiveMQ to populate a relational reporting database that exists independently at each datacenter. I'll give you a little more background. My team (the data engineering team) did not implement the configuration, We have been brought in because of some performance issues, etc. and it appears that since it has KahaDB, the powers that be assume that it's a database and should fall under our team. In any event, we have been researching ActiveMQ and our configuration to help out. I believe that the thought process was that the apps would work as follows: They would produce to the topic where there are 5 "streams", which is what our company calls each individual application server, and the application would open 25 threads per topic, there are 10 topics, essentially one topic for each message type. So we're looking at 1250 threads open producing messages to ActiveMQ. I assume they used a topic and then used the camel routes to intercept those messages and write them to the local and local remote queues as we have the same number of threads consuming messages from the local queue and then 25 consumers set on the camel routes that route the messages from the local remote queue to the opposite datacenter. We need to be able to consume messages in parallel and it doesn't appear that is possible using a topic in ActiveMQ 5.X as it is JMS 1.1 compliant. It appears that Virtual Topics are used to get around this. Correct me if I am wrong on that. Are you suggesting that we could use a "network of brokers" with Virtual Topics to still get the functionality we have with the camel routes? William On Sun, Sep 22, 2019 at 1:39 AM Tim Bain <tb...@alumni.duke.edu> wrote: > William, > > If I understand you correctly, you're using Camel to clone the message so > you get one per data center, and then using Camel to move the message from > the data center at which it was produced to the other one. And you're doing > this so that you can have all messages reach consumers in both data > centers, though I'm a little confused about why you're routing messages > sent to topics (with pub/sub semantics, supporting multiple consumers) to a > single local queue in each data center (which would allow only one consumer > per data center to process the message). Did I get that right? > > If so, it does seem like a network of brokers would handle this natively > without all the Camel routes. You'd ensure that messages and subscriptions > for the topic in question is forwarded across the network bridge, and all > consumers in either data center would get delivered a copy of the message. > What is it that you're accomplishing with the Camel routes that would not > be accomplished with this setup? > > Tim > > On Tue, Sep 17, 2019 at 11:11 AM William Culler <willcul...@gmail.com> > wrote: > > > We have the following configuration. > > > > 3 brokers in Datacenter A in a master/slave configuration using a shared > > file system. 3 brokers in Datacenter B in a master/slave configuration > > using a shared file system. > > > > There are producers sending messages to topics at Datacenter A. There are > > Camel routes routing those messages from the topics to separate local > > queues, 1 for local processing and 1 for remote processing, where another > > set of Camel routes route the messages from the local remote queue to > > remote queues at Datacenter B. > > > > The same configuration exists at Datacenter B where the routing for the > > remote works in the opposite direction to Datacenter A. > > > > This configuration is intended to give us an active/active > multi-datacenter > > cluster. > > > > Even though this configuration seems to work, we are wondering if this > > would be the recommended way to do this. I am thinking in terms of using > > "network of brokers", but reading the documentation, it doesn't seem like > > that would accomplish what we are accomplishing with the Camel routes. > > > > We do have problems from time to time, but we do not know for sure if the > > Camel routes are the cause. > > > > Any insight would be appreciated. > > >
