Hello Karan. I agree the initial experience could be a lot friendlier. Most of the complexity there is inherited from Connect, but it's compounded when multiple clusters are involved.
I don't think we want to change Connect's (or MM2's) defaults to assume a single broker cluster -- it'd be too easy to overlook such defaults in a prod environment, which could be dangerous. It might be nice to add a high-level internal.replication.factor property that, if specified, overrides the replication factor for all internal topics. I actually had a prototype of this at one point but removed it, as it wasn't part of the KIP. I agree we should modify the example config to work out-of-the-box with single-broker clusters, and just call out that these should not be used in a production environment. Lemme know if you want to create a PR or if you'd like me to. It looks like you've already done most of the work :) Ryanne On Tue, Dec 24, 2019, 3:41 AM Karan Kumar <karankumar1...@gmail.com> wrote: > Hi > > One of the nice things about kafka is setting up in the local environment > is really simple. I was giving a try to the latest feature ie MM2 and found > it took me some time to get a minimal setup running. > Default config provided assumes that there will already be 3 brokers > running due to the default replication factor of the admin topics the mm2 > connector creates. > > This got me thinking that most of the people would follow the same approach > I followed. > 1. Start a single broker cluster on 9092 > 2. Start another single cluster broker on, let's say, 10002 > 3. Start mm2 by"./bin/connect-mirror-maker.sh > ./config/connect-mirror-maker.properties" > > What happened was I had to supply a lot more configs > > clusters = A, B > > # connection information for each cluster > A.bootstrap.servers = localhost:9092 > B.bootstrap.servers = localhost:10092 > > # enable and configure individual replication flows > A->B.enabled = * > A->B.topics = test > B->A.enabled = true > B->A.topics = * > > > A.heartbeats.topic.replication.factor=1 > A.checkpoints.topic.replication.factor=1 > A.offset-syncs.topic.replication.factor=1 > > B.heartbeats.topic.replication.factor=1 > B.checkpoints.topic.replication.factor=1 > B.offset-syncs.topic.replication.factor=1 > > > A.offset.storage.replication.factor=1 > B.offset.storage.replication.factor=1 > > > > A.status.storage.replication.factor=1 > B.status.storage.replication.factor=1 > > > A.config.storage.replication.factor=1 > B.config.storage.replication.factor=1 > > > The server.properties has bunch of properties like > "offsets.topic.replication.factor=1 > transaction.state.log.replication.factor=1" > which make it easier for people to start a local single broker kafka > cluster. > > > Does it make sense to have a similar config as the default mirror maker > config so that it becomes easier for people to use the MM2 feature. > > -- > Thanks > Karan >
