Hello, We have a use case where we want to replicate an event stream which exists outside of kafka into a kafka topic (single partition). The event stream has sequence ids which always increase by 1. We want to preserve this ordering.
The difficulty is that we want to be able to have the process that writes these events automatically fail-over if it dies. While ZooKeeper can guarantee a single writer at a given point in time we are worried about delayed network packets, bugs and long GC pauses. One solution we've thought of is to set the sequence_id as the key for the Kafka messages and have a proxy running on each Kafka broker which refuses to write new messages if they don't have the next expected key. This seems to solve any issue we would have with badly behaving networks or processes. Is there a better solution? Should we just handle these inconsistencies in our consumers? Are we being too paranoid? As a side-note, it seems like this functionality (guaranteeing that all keys in a partition are in sequence on a particular topic) may be a nice option to have in Kafka proper. Thanks, Bob
