Hey Nitin, it's not uncommon to use separate clusters and MirrorMaker for this purpose, e.g. separate ingest and ETL clusters. That way your ETL cluster has only one producer (MM) and only one consumer (Camus).
It's also possible you just need more brokers and/or more partitions. Ryanne On Sat, May 23, 2020, 2:38 AM nitin agarwal <nitingarg...@gmail.com> wrote: > Hi All, > > We have Camus consumer which reads data from Kafka and writes to HDFS. > Whenever this consumer runs, there is impact on producer latencies. We > don't want producer latencies to be increased in anyway. > > I want to know which design pattern is more suited for supporting this use > case. > 1) Single Kafka cluster with efficient throttling limits for consumers. > 2) Separate Kafka cluster for producers and consumers. Data is replicated > using MirrorMaker. > > Thanks, > Nitin >
