Hi Ryanne Thanks for the response. Based on our discussion, raised a jira : KAFKA-9337 <https://issues.apache.org/jira/browse/KAFKA-9337>. Please find the PR here <https://github.com/apache/kafka/pull/7872>.
On Tue, Dec 24, 2019 at 8:18 PM Ryanne Dolan <ryannedo...@gmail.com> wrote: > Hello Karan. I agree the initial experience could be a lot friendlier. Most > of the complexity there is inherited from Connect, but it's compounded when > multiple clusters are involved. > > I don't think we want to change Connect's (or MM2's) defaults to assume a > single broker cluster -- it'd be too easy to overlook such defaults in a > prod environment, which could be dangerous. It might be nice to add a > high-level internal.replication.factor property that, if specified, > overrides the replication factor for all internal topics. I actually had a > prototype of this at one point but removed it, as it wasn't part of the > KIP. > > I agree we should modify the example config to work out-of-the-box with > single-broker clusters, and just call out that these should not be used in > a production environment. Lemme know if you want to create a PR or if you'd > like me to. It looks like you've already done most of the work :) > > Ryanne > > > On Tue, Dec 24, 2019, 3:41 AM Karan Kumar <karankumar1...@gmail.com> > wrote: > > > Hi > > > > One of the nice things about kafka is setting up in the local environment > > is really simple. I was giving a try to the latest feature ie MM2 and > found > > it took me some time to get a minimal setup running. > > Default config provided assumes that there will already be 3 brokers > > running due to the default replication factor of the admin topics the mm2 > > connector creates. > > > > This got me thinking that most of the people would follow the same > approach > > I followed. > > 1. Start a single broker cluster on 9092 > > 2. Start another single cluster broker on, let's say, 10002 > > 3. Start mm2 by"./bin/connect-mirror-maker.sh > > ./config/connect-mirror-maker.properties" > > > > What happened was I had to supply a lot more configs > > > > clusters = A, B > > > > # connection information for each cluster > > A.bootstrap.servers = localhost:9092 > > B.bootstrap.servers = localhost:10092 > > > > # enable and configure individual replication flows > > A->B.enabled = * > > A->B.topics = test > > B->A.enabled = true > > B->A.topics = * > > > > > > A.heartbeats.topic.replication.factor=1 > > A.checkpoints.topic.replication.factor=1 > > A.offset-syncs.topic.replication.factor=1 > > > > B.heartbeats.topic.replication.factor=1 > > B.checkpoints.topic.replication.factor=1 > > B.offset-syncs.topic.replication.factor=1 > > > > > > A.offset.storage.replication.factor=1 > > B.offset.storage.replication.factor=1 > > > > > > > > A.status.storage.replication.factor=1 > > B.status.storage.replication.factor=1 > > > > > > A.config.storage.replication.factor=1 > > B.config.storage.replication.factor=1 > > > > > > The server.properties has bunch of properties like > > "offsets.topic.replication.factor=1 > > transaction.state.log.replication.factor=1" > > which make it easier for people to start a local single broker kafka > > cluster. > > > > > > Does it make sense to have a similar config as the default mirror maker > > config so that it becomes easier for people to use the MM2 feature. > > > > -- > > Thanks > > Karan > > > -- Thanks Karan
