Re: Kafka transaction between 2 kafka clusters

2019-05-10 Thread Kamal Chandraprakash
MirrorMaker 2.0 stores the offsets of one cluster in another. So, you can read the offsets from the same cluster once this KIP is implemented. https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-RemoteClusterUtils On Fri, May 10, 2019 at 12:29 PM Em

Re: Kafka transaction between 2 kafka clusters

2019-05-09 Thread Emmanuel
Thanks! I wonder if this is a bit far-fetched as no one seems to do this at the moment. On Fri, May 10, 2019 at 12:50 AM Guozhang Wang wrote: > Hello Emmanuel, > > Yes I think it is do-able technically. Note that it means the offsets of > cluster A would be stored on cluster B and hence upon res

Re: Kafka transaction between 2 kafka clusters

2019-05-09 Thread Guozhang Wang
Hello Emmanuel, Yes I think it is do-able technically. Note that it means the offsets of cluster A would be stored on cluster B and hence upon restarting one need to talk to cluster B in order to get the committed position in cluster A. Guozhang On Thu, May 9, 2019 at 11:58 AM Emmanuel wrote:

Kafka transaction between 2 kafka clusters

2019-05-09 Thread Emmanuel
Hello, I would like to know if there is a Java client that would allow me to consume from topics on a cluster A and produce to topics on a cluster B with exactly-once semantics. My understanding of the Kafka transactions is that on the paper it could work, but the kafka java client assumes both ar