Thanks Tim. We're going to try a couple different solutions.
On Fri, Sep 27, 2019 at 1:13 AM Tim Bain <tb...@alumni.duke.edu> wrote: > Yes, if you're looking for competing consumer semantics but for each > message to be consumed exactly once in each data center, then a network of > brokers with a virtual topic atop the two queues would be my first thought > for how to do this. I've not personally done any bridging of virtual > destinations across a NoB, but I think that as long as you forward the > queues between the brokers and don't forward the topic, it should work as > your current setup does without the need for any Camel configuration. > > Tim > > On Mon, Sep 23, 2019 at 10:21 PM William Culler <willcul...@gmail.com> > wrote: > > > Hello Tim. > > > > Thanks for taking the time for a response. > > > > You are correct. The setup is being used to keep the 2 datacenters > > completely in sync. It allows us to switch our reporting frontend to the > > other datacenter at any time if we are having issues at the opposite > > datacenter. The messages are consumed from ActiveMQ to populate a > > relational reporting database that exists independently at each > datacenter. > > > > I'll give you a little more background. My team (the data engineering > > team) did not implement the configuration, We have been brought in > because > > of some performance issues, etc. and it appears that since it has KahaDB, > > the powers that be assume that it's a database and should fall under our > > team. > > > > In any event, we have been researching ActiveMQ and our configuration to > > help out. > > > > I believe that the thought process was that the apps would work as > follows: > > > > They would produce to the topic where there are 5 "streams", which is > what > > our company calls each individual application server, and the application > > would open 25 threads per topic, there are 10 topics, essentially one > topic > > for each message type. So we're looking at 1250 threads open producing > > messages to ActiveMQ. I assume they used a topic and then used the camel > > routes to intercept those messages and write them to the local and local > > remote queues as we have the same number of threads consuming messages > from > > the local queue and then 25 consumers set on the camel routes that route > > the messages from the local remote queue to the opposite datacenter. We > > need to be able to consume messages in parallel and it doesn't appear > that > > is possible using a topic in ActiveMQ 5.X as it is JMS 1.1 compliant. It > > appears that Virtual Topics are used to get around this. Correct me if > I > > am wrong on that. > > > > Are you suggesting that we could use a "network of brokers" with Virtual > > Topics to still get the functionality we have with the camel routes? > > > > William > > > > On Sun, Sep 22, 2019 at 1:39 AM Tim Bain <tb...@alumni.duke.edu> wrote: > > > > > William, > > > > > > If I understand you correctly, you're using Camel to clone the message > so > > > you get one per data center, and then using Camel to move the message > > from > > > the data center at which it was produced to the other one. And you're > > doing > > > this so that you can have all messages reach consumers in both data > > > centers, though I'm a little confused about why you're routing messages > > > sent to topics (with pub/sub semantics, supporting multiple consumers) > > to a > > > single local queue in each data center (which would allow only one > > consumer > > > per data center to process the message). Did I get that right? > > > > > > If so, it does seem like a network of brokers would handle this > natively > > > without all the Camel routes. You'd ensure that messages and > > subscriptions > > > for the topic in question is forwarded across the network bridge, and > all > > > consumers in either data center would get delivered a copy of the > > message. > > > What is it that you're accomplishing with the Camel routes that would > not > > > be accomplished with this setup? > > > > > > Tim > > > > > > On Tue, Sep 17, 2019 at 11:11 AM William Culler <willcul...@gmail.com> > > > wrote: > > > > > > > We have the following configuration. > > > > > > > > 3 brokers in Datacenter A in a master/slave configuration using a > > shared > > > > file system. 3 brokers in Datacenter B in a master/slave > configuration > > > > using a shared file system. > > > > > > > > There are producers sending messages to topics at Datacenter A. There > > are > > > > Camel routes routing those messages from the topics to separate local > > > > queues, 1 for local processing and 1 for remote processing, where > > another > > > > set of Camel routes route the messages from the local remote queue to > > > > remote queues at Datacenter B. > > > > > > > > The same configuration exists at Datacenter B where the routing for > the > > > > remote works in the opposite direction to Datacenter A. > > > > > > > > This configuration is intended to give us an active/active > > > multi-datacenter > > > > cluster. > > > > > > > > Even though this configuration seems to work, we are wondering if > this > > > > would be the recommended way to do this. I am thinking in terms of > > using > > > > "network of brokers", but reading the documentation, it doesn't seem > > like > > > > that would accomplish what we are accomplishing with the Camel > routes. > > > > > > > > We do have problems from time to time, but we do not know for sure if > > the > > > > Camel routes are the cause. > > > > > > > > Any insight would be appreciated. > > > > > > > > > >
