Re: Kafka 0.9 MirrorMaker failing with Batch Expired when producing to Kafka 1.0 cluster

2018-03-13 Thread Andrew Otto
​Hm, the hardware should be mostly beefier in this new cluster, but there are a couple of differences: Mainly we are now using RAID instead of JBOD, and​ our high volume producers (that 150K / second I mentioned) use TLS. Also, other producers seem have no problems, it is only MirrorMaker. Even s

Re: Kafka 0.9 MirrorMaker failing with Batch Expired when producing to Kafka 1.0 cluster

2018-03-13 Thread Andras Beni
Hi Andrew, It seems the throughput of the new cluster is smaller than that of the old cluster. And for this reason MirrorMaker cannot send messages fast enough (i.e. they expire). I recommend comparing the configurations. For the hanging MirrorMaker instances, I think looking at stack dumps would

Kafka 0.9 MirrorMaker failing with Batch Expired when producing to Kafka 1.0 cluster

2018-03-12 Thread Andrew Otto
Hi all, I’m troubleshooting a MirrorMaker issue, and am not quite sure yet why this is happening, so I’d thought I’d ask here in case anyone else has seen this before. We’ve been running a Kafka 1.0 cluster for a few months now, replicating data from a Kafka 0.9.0.1 cluster using 0.9.0.1 MirrorMa