I don't know any other way that manually synching zookeeper ensembles plus brokers in the new world :(
In my example below, I have three node Zookeeper ensemble with 9 Kafka brokers [image: image.png] First of all the zookeeper conf file under $ZOOKEEPER_HOME/conf/zoo.cfg has to match tickTime=2000 dataDir=/data6/hduser/zookeeper/zookeeper nitLimit=5 syncLimit=2 # the port at which the clients will connect clientPort=2181 server.1=rhes75:2888:3888 server.2=rhes564:2888:3888 server.3=rhes76:2888:3888 Those server entries need to reflect new environment Also anything under dataDir needs to be synched for each zookeeper For Kafka I have three brokers on each host. under directory $KAFKA_HOME/config, there are three server properties file. You need to copy these to the new environment and modify them accordingly. By default there is one log partition for topic The log directory for each properties file needs to be copied across and kept in sync Example, I have three of them server1.properties:log.dirs=/tmp/kafka-logs-1 server2.properties:log.dirs=/tmp/kafka-logs-2 server3.properties:log.dirs=/tmp/kafka-logs-3 I have not done this before but I guess if you have some clustering and keep-in-sync mechanism, then it is possible to switch over. Otherwise a downtime will be required. Please also note that running $ZOOKEEPER_HOME/bin/zkCli.sh $ZOOKEEPER_HOME/bin/zkCli.sh Connecting to localhost:2181 Welcome to ZooKeeper! [zk: localhost:2181(CONNECTED) 0] ls / [cluster, controller_epoch, controller, brokers, zookeeper, admin, isr_change_notification, consumers, log_dir_event_notification, latest_producer_id_block, config, hbase] [zk: localhost:2181(CONNECTED) 1] ls /brokers/topics [mdout, final, test, __consumer_offsets, md] you can get list of topics etc. I think it will be involved unless someone else has better ideas. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Thu, 11 Jul 2019 at 14:00, Elliot West <tea...@gmail.com> wrote: > Thanks for your reply. > > Yes, you can assume that nothing is shared between clusters. > > There is no specific topic. I'm trying to establish general patterns that > can be applied to achieve this. > > Thanks, > > Elliot. > > On Thu, 11 Jul 2019 at 10:50, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > > > Hi Elliot, > > > > As you are moving the topic from one cluster to another, I assume it > > implies a new zookeeper ensemble plus sets of new brokers? > > > > Can you describe the current topic? > > > > ${KAFKA_HOME}/bin/kafka-topics.sh --describe --zookeeper <HOST:PORT>, > > <HOST:PORT>, --topic <TOPIC> > > > > HTH > > > > Dr Mich Talebzadeh > > > > > > > > LinkedIn * > > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > < > > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > >* > > > > > > > > http://talebzadehmich.wordpress.com > > > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > > loss, damage or destruction of data or any other property which may arise > > from relying on this email's technical content is explicitly disclaimed. > > The author will in no case be liable for any monetary damages arising > from > > such loss, damage or destruction. > > > > > > > > > > On Thu, 11 Jul 2019 at 09:50, Elliot West <tea...@gmail.com> wrote: > > > > > Hello, > > > > > > I like to understand what strategies can be applied to migrate events, > > > producers, and consumers from one topic to another. Typically I'm > > thinking > > > of cases where: > > > > > > > > > - we wish to migrate a topic from one cluster to another > > > - high availability - minimise amount of time topic is unavailable > to > > > producers/consumers > > > - no data loss (at least once) > > > - avoid excessive latency (keep events flowing - no 'stopping the > > > world') > > > - preserve event ordering wrt to partitions > > > - there is a significant retention interval (days) that we wish to > > move > > > to the destination topic also > > > > > > > > > I can imagine that tools such as MM1+2, Replicator, etc. are useful in > > > these circumstances. However, I'm more interested in orchestration > > > patterns: what things need to be moved, in what order, and subject to > > what > > > conditions? > > > > > > Thanks for your time, > > > > > > Elliot. > > > > > >
