One could refactor MirrorMaker to commit the source cluster's offset in the
target cluster's instead (in a special topic)
This would technically allow achieving exactly once using the Transactional
API.
But there's work associated with that
Let me know if I’m missing something
On 12/1/18, 6:15 am, "Matthias J. Sax" <matth...@confluent.io> wrote:
No.
Transactions are designed to work within a single cluster, not cross
cluster, ie, if you have a read-process-write pattern similar to what
Kafka Streams does.
-Matthias
On 1/11/18 12:46 AM, Jiri Humpolicek wrote:
> Hi Everyone,
>
> since kafka 0.11.x supports exactly-once semantics, I want to be sure,
that
> it is possible to achieve exactly-once delivery across kafka clusters
using
> MirrorMaker.
>
> We have got two locations with "primary" cluster in each location and for
> each location we have got one "aggregation" cluster which mirrors data
from
> all primary clusters.
>
> Currently we deduplicate messages after copying data from aggregation
kafka
> to HDFS by separete YARN application. But in aggregation kafka duplicates
> remains. So I want to ensure that there are no duplicates and data loss
in
> kafka as well. In this case our deduplication yarn application could not
be
> use anymore.
>
> If it is possible, how to configure MirrorMaker to achieve exactly-once
> delivery across primary and aggregation clusters?
>
>
> Thanks and have a nice day, Jiri Humpolicek
>
