No you can't because: - because of producer failures some messages may be duplicated. - your not sure the cluster your copying from hasn't some already removed data.
We try to solve the same problem, and are probably going to solve it by copying the timestamps with the mirror maker, and on the switch let the consumer go near the and, and check with the timestamp when it needs to start processing again. On Thu, Jun 2, 2016 at 5:10 AM Dave Cahill <dpcah...@gmail.com> wrote: > Hi, > > I've read up a little on MirrorMaker (e.g. the wiki [1] and KIP-3 [2]), but > haven't yet found a definitive answer to the following question. > > Let's assume I am producing a certain topic to a Kafka cluster in > datacenter A. > > I set up MirrorMaker in datacenter B and C to mirror the topic from > datacenter A. > > Can I assume that the mirrored data in datacenter B and C are exactly the > same, including the same offsets? For example, let's say a consumer in > datacenter B dies, and I know the offset up to which it has read. Can a > consumer in datacenter C take over exactly where B left off by reading from > its own copy starting at the same offset? If MirrorMaker has logic to > prevent dupes (something like what's described in [3]) and lost messages, > it seems like this should work. > > Please let me know if my terminology is imprecise and I'll try to clarify! > > Thanks, > Dave. > > > [1] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330 > [2] > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-3+-+Mirror+Maker+Enhancement > [3] > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIgetexactly-oncemessagingfromKafka > ? >
