As org.kaliy.kafka.connect.rss.RssSourceConnector is open-source, it seems a better idea to extend it to read from multiple RSS URI, so you may still enjoy the fine-granularity of automated management of a set of URI within one connector, while not creating too many connectors
On 2020/12/09 16:17:39, Alberto Coello <coloalcol...@gmail.com> wrote: > So, I have the intention to deploy a lot of connectors(An unbounded number, > like hundreds, thousands or more) in my Kafka Connect cluster, each of them > would be of the same class(org.kaliy.kafka.connect.rss.RssSourceConnector) > but with different configuration, the difference would be in the RSS URI > they read from. I mean to create, and manage, them in an automated way > implementing a process in charge of managing their status via Connect REST > API. BUT as all Connect configuration is saved in three > topics: config.storage.topic, offset.storage.topic and > status.storage.topic; I am doubting about its capacity to handle this > number of connectors. Any other problems you can higlight about this > approach would be appreciated, thanks a lot!. >
