Not an expert, but that sounds like a very reasonable use case for Kafka. The log.retention.* configs on the edge brokers should cover your TTL needs.
On Thu, Jan 14, 2016 at 3:37 PM, Jason J. W. Williams < jasonjwwilli...@gmail.com> wrote: > Hello, > > We historically have been a RabbitMQ environment, but we're looking at > using Kafka for a new project and I'm wondering if the following > topology/setup would work well in Kafka (for RMQ we'd use federation): > > * Multiple remote datacenters consisting each of a single server running an > HTTP application that receives client data and generates events. Each > server would also run single-node Kafka "cluster". The application would > write events as messages into the single-node Kafka "cluster" running on > the same machine. > * A hub datacenter that the remote data centers are connected to via SSL. > The hub data center would run a multi-node Kafka cluster (3 nodes). > * Use mirrormaker in the hub data center to mirror event messages from each > of the remote single-node servers into the hub's central Kafka cluster, > where all of the real consumers are listening. > > The problem set is each of the remote servers is collecting data from > customers over HTTP and returning responses, but those remote servers are > also generating events from those customer interactions. We want to publish > those events into a central hub data center for analytics. We want the > event messages at the remote servers to queue up when their network > connections to the hub data center is unreliable, and automatically relay > queued messages to the hub data center when the network comes back...making > the event relay system tolerant to WAN network faults. We'd also want to > set up some kind of TTL on queued messages, so if the WAN connection to the > hub is down for an extended period of time, the messages queued on the > remote servers don't build up infinitely. > > Any thoughts on if this setup is advisable/inadvisable with Kafka (or any > other thoughts on it) would be greatly appreciated. > > -J >
