Hello, We historically have been a RabbitMQ environment, but we're looking at using Kafka for a new project and I'm wondering if the following topology/setup would work well in Kafka (for RMQ we'd use federation):
* Multiple remote datacenters consisting each of a single server running an HTTP application that receives client data and generates events. Each server would also run single-node Kafka "cluster". The application would write events as messages into the single-node Kafka "cluster" running on the same machine. * A hub datacenter that the remote data centers are connected to via SSL. The hub data center would run a multi-node Kafka cluster (3 nodes). * Use mirrormaker in the hub data center to mirror event messages from each of the remote single-node servers into the hub's central Kafka cluster, where all of the real consumers are listening. The problem set is each of the remote servers is collecting data from customers over HTTP and returning responses, but those remote servers are also generating events from those customer interactions. We want to publish those events into a central hub data center for analytics. We want the event messages at the remote servers to queue up when their network connections to the hub data center is unreliable, and automatically relay queued messages to the hub data center when the network comes back...making the event relay system tolerant to WAN network faults. We'd also want to set up some kind of TTL on queued messages, so if the WAN connection to the hub is down for an extended period of time, the messages queued on the remote servers don't build up infinitely. Any thoughts on if this setup is advisable/inadvisable with Kafka (or any other thoughts on it) would be greatly appreciated. -J
