Yes, if you're looking for competing consumer semantics but for each message to be consumed exactly once in each data center, then a network of brokers with a virtual topic atop the two queues would be my first thought for how to do this. I've not personally done any bridging of virtual destinations across a NoB, but I think that as long as you forward the queues between the brokers and don't forward the topic, it should work as your current setup does without the need for any Camel configuration.
Tim On Mon, Sep 23, 2019 at 10:21 PM William Culler <willcul...@gmail.com> wrote: > Hello Tim. > > Thanks for taking the time for a response. > > You are correct. The setup is being used to keep the 2 datacenters > completely in sync. It allows us to switch our reporting frontend to the > other datacenter at any time if we are having issues at the opposite > datacenter. The messages are consumed from ActiveMQ to populate a > relational reporting database that exists independently at each datacenter. > > I'll give you a little more background. My team (the data engineering > team) did not implement the configuration, We have been brought in because > of some performance issues, etc. and it appears that since it has KahaDB, > the powers that be assume that it's a database and should fall under our > team. > > In any event, we have been researching ActiveMQ and our configuration to > help out. > > I believe that the thought process was that the apps would work as follows: > > They would produce to the topic where there are 5 "streams", which is what > our company calls each individual application server, and the application > would open 25 threads per topic, there are 10 topics, essentially one topic > for each message type. So we're looking at 1250 threads open producing > messages to ActiveMQ. I assume they used a topic and then used the camel > routes to intercept those messages and write them to the local and local > remote queues as we have the same number of threads consuming messages from > the local queue and then 25 consumers set on the camel routes that route > the messages from the local remote queue to the opposite datacenter. We > need to be able to consume messages in parallel and it doesn't appear that > is possible using a topic in ActiveMQ 5.X as it is JMS 1.1 compliant. It > appears that Virtual Topics are used to get around this. Correct me if I > am wrong on that. > > Are you suggesting that we could use a "network of brokers" with Virtual > Topics to still get the functionality we have with the camel routes? > > William > > On Sun, Sep 22, 2019 at 1:39 AM Tim Bain <tb...@alumni.duke.edu> wrote: > > > William, > > > > If I understand you correctly, you're using Camel to clone the message so > > you get one per data center, and then using Camel to move the message > from > > the data center at which it was produced to the other one. And you're > doing > > this so that you can have all messages reach consumers in both data > > centers, though I'm a little confused about why you're routing messages > > sent to topics (with pub/sub semantics, supporting multiple consumers) > to a > > single local queue in each data center (which would allow only one > consumer > > per data center to process the message). Did I get that right? > > > > If so, it does seem like a network of brokers would handle this natively > > without all the Camel routes. You'd ensure that messages and > subscriptions > > for the topic in question is forwarded across the network bridge, and all > > consumers in either data center would get delivered a copy of the > message. > > What is it that you're accomplishing with the Camel routes that would not > > be accomplished with this setup? > > > > Tim > > > > On Tue, Sep 17, 2019 at 11:11 AM William Culler <willcul...@gmail.com> > > wrote: > > > > > We have the following configuration. > > > > > > 3 brokers in Datacenter A in a master/slave configuration using a > shared > > > file system. 3 brokers in Datacenter B in a master/slave configuration > > > using a shared file system. > > > > > > There are producers sending messages to topics at Datacenter A. There > are > > > Camel routes routing those messages from the topics to separate local > > > queues, 1 for local processing and 1 for remote processing, where > another > > > set of Camel routes route the messages from the local remote queue to > > > remote queues at Datacenter B. > > > > > > The same configuration exists at Datacenter B where the routing for the > > > remote works in the opposite direction to Datacenter A. > > > > > > This configuration is intended to give us an active/active > > multi-datacenter > > > cluster. > > > > > > Even though this configuration seems to work, we are wondering if this > > > would be the recommended way to do this. I am thinking in terms of > using > > > "network of brokers", but reading the documentation, it doesn't seem > like > > > that would accomplish what we are accomplishing with the Camel routes. > > > > > > We do have problems from time to time, but we do not know for sure if > the > > > Camel routes are the cause. > > > > > > Any insight would be appreciated. > > > > > >
