Thanks for the quick reply. This may work but I have like 5 topics to
listen to right now, I am trying to keep all topics in an array in a
properties file and trying to read all at once. This way it is dynamic and
you have one code block like below and you may add or delete topics from
the config file without changing code. If someone confirms that it does not
work, I would have to do something like you have provided.
val df_cluster1 = spark
.read
.format("kafka")
.option("kafka.bootstrap.servers",
"cluster1_host:cluster1_port,cluster2_host:port")
.option("subscribe", "topic1, topic2,topic3,topic4,topic5")
